#2019-07-1414:39ikitommiAny comments and critique on design welcome#2019-08-0115:24ikitommi(defn composite-perf2 []
(let [tests [[true [-1]]
[true [-1 1 2]]
[false [-1 0 2]]
[false [-1 -1 -1 -1]]]
assert! (fn [f]
(doseq [[expected data] tests]
(assert (= expected (f data)))))]
;; 155ns
(let [valid? (fn [x]
(and (vector? x)
(<= (count x) 3)
(every? #(and (int? %) (or (pos-int? %) (neg-int? %))) x)))]
(assert! valid?)
(cc/quick-bench
(valid? [-1 1 2])))
;; 27ns
(let [valid? (m/validator
[:vector {:max 3}
[:and int? [:or pos-int? neg-int?]]])]
(assert! valid?)
(cc/quick-bench
(valid? [-1 1 2])))
;; 506ns
(let [spec (s/coll-of
(s/and int? (s/or :pos-int pos-int? :neg-int neg-int?))
:kind vector?
:max-count 3)
valid? (partial s/valid? spec)]
(assert! valid?)
(cc/quick-bench
(valid? [-1 1 2])))))
#2019-08-0209:02ikitommisome discussion about defaults: https://github.com/metosin/malli/issues/25#2019-10-0319:17bedershey there, I was trying to play with malli a bit today#2019-10-0319:17bedershaving trouble downloading its dependencies though#2019-10-0319:17bedersI used the latest commit id and `{:git/url "https://github.com/metosin/malli"
:sha "59b226968025502e3830f5c6404b0f95f8c080e6"}`#2019-10-0319:18bedersNot super familiar with deps I must admit#2019-10-0319:18bedersgetting this error at the moment:
Error building classpath. Unable to compare versions for borkdude/edamame: {:mvn/version "0.0.5-SNAPSHOT", :deps/manifest :mvn} and {:git/url "", :sha "b577e565b136d3dd51945fe874049d4297946f57", :deps/manifest :deps, :deps/root "/Users/beders/.gitlibs/libs/borkdude/edamame/b577e565b136d3dd51945fe874049d4297946f57"}
#2019-10-0319:45ikitommi@beders hmm.. not sure what’s wrong with edamame, but this should work:
clj -Sdeps '{:deps {metosin/malli {:git/url "" :sha "10b09bef72a52755764ba21933edc983fc4135e7"}, borkdude/edamame {:git/url "", :sha "d7d0f12c336dc513702c8c704b147223f85e377f"}}}'
#2019-10-0319:45bedersthank you!#2019-10-0511:07ikitommiasked in #tools-deps how to resolve that, meanwhile, changed edamame to use mvn versio, so this works now:
clj -Sdeps '{:deps {metosin/malli {:git/url "" :sha "22122b2ff9434bfd0db8d72fcd06cb48914c6eb4"}}}'
#2019-10-0818:49sammikkohi, couldnt find channel for jsonista.. is it possible to de-serialize json date strings "2019-10-04T09:08:36.078Z" to org.joda.time.DateTime objects, based on the format of the incoming string... i understand it works the other way around#2019-10-0818:52sammikkoi'd just like to get a clojure map from json string, but the strings looking like dates should be converted into DateTime..#2019-10-0820:09ikitommiyou need to describe which fields should be dates and apply coercion with that. E.g. need to use spec/schema or malli.#2019-10-0820:10ikitommibased on format... you could also roll your own walker that inspects the strings, but would be slow I think#2019-10-0907:28sammikkoyeah.. we are using Schema atm and schema bijections library to do the conversions.. but its very slow..#2019-10-0907:38ikitommiSchema coercion should be fast? Malli will be Really Fast, but doesn't have an easy way to create custom transforming schemas. One PR away from....#2019-10-0907:58sammikkothanks.. I'll check out coercion as well..#2019-10-0908:26ikitommiok, could be faster. the date parsing on java-side seems quite slow in general:
(require '[malli.core :as m])
(require '[malli.transform :as mt])
(def Pizza
[:map
[:id int?]
[:name string?]
[:created inst?]])
(def json->Pizza
(m/transformer
Pizza
mt/json-transformer))
(json->Pizza
{:id 1
:name "salami"
:created "2019-10-04T09:08:36.078Z"})
;{:id 1
; :name "salami"
; :created #inst"2019-10-04T09:08:36.078-00:00"}
(require '[criterium.core :as cc])
(cc/quick-bench
(json->Pizza
{:id 1
:name "salami"
:created "2019-10-04T09:08:36.078Z"}))
; Evaluation count : 386256 in 6 samples of 64376 calls.
; Execution time mean : 1.719360 µs
; Execution time std-deviation : 100.090402 ns
; Execution time lower quantile : 1.601860 µs ( 2.5%)
; Execution time upper quantile : 1.811632 µs (97.5%)
; Overhead used : 1.862824 ns
#2019-10-0908:29ikitommioh, Plumatic Schema doesn’t have date coercion oob, forgot about that.#2019-10-0908:32ikitommiwith schema-tools:
(require '[schema.core :as s])
(require '[schema-tools.coerce :as stc])
(def json->Pizza2
(stc/coercer {:id s/Int, :name s/Str, :created s/Inst} stc/json-coercion-matcher))
(json->Pizza2
{:id 1
:name "salami"
:created "2019-10-04T09:08:36.078Z"})
;{:id 1
; :name "salami"
; :created #inst"2019-10-04T09:08:36.078-00:00"}
(cc/quick-bench
(json->Pizza2
{:id 1
:name "salami"
:created "2019-10-04T09:08:36.078Z"}))
; Evaluation count : 261396 in 6 samples of 43566 calls.
; Execution time mean : 2.352737 µs
; Execution time std-deviation : 173.369968 ns
; Execution time lower quantile : 2.255370 µs ( 2.5%)
; Execution time upper quantile : 2.636725 µs (97.5%)
; Overhead used : 1.600826 ns
#2019-10-2517:19sammikkohey @ikitommi what you think of specter lib.. i know you like performance etc., you guys like to use it? see any trouble with it?#2019-10-2517:23sammikkoI get some 90s flashbacks of the XSLT and xpath times, but it sure seems convenient..#2019-10-2813:06eskosHey so I feel really dumb but how would I make a malli schema which validates that I have a non-empty string 😅 I’m thinking something like [:and string? [:not= blank?]] but there’s no blank? predicate support…#2019-10-2813:06eskosblank? does string check internally as well so it doesn’t need to be :and schema but anyhoos…#2019-10-2813:54ikitommi@suomi.esko You can use any predicate with :fn, either: [:fn str/blank?] (local, fast) or [:fn 'str/blank?] (portable)#2019-10-2814:05eskos@ikitommi Seems to work for the positive case, but I really do want to test the negation of that and I can’t seem to get it to work, or at least :not= isn’t what I should use. I think there isn’t a complementing predicate yet? 🙂#2019-10-2814:07ikitommi[:and string? some?] or [:fn {:error/message "should be non-empty string"} '(complement str/blank?)]?#2019-10-2814:07ikitommicould be :not ....#2019-10-2814:09eskoslatter works though#2019-10-2814:11eskosformer indeed does not, some? is effectively “not nil” check#2019-10-3013:21eskos@ikitommi Had a thought, should I create an issue on GitHub about my “non-empty string” problem or do you think this is eventually solved as side effect or by other means?#2019-10-3014:03ikitommiissue welcome, and suggestion how to fix that 🙂#2019-10-3014:03ikitommihttps://malli.io/?value=%22%22&schema=%5B%3Aand%20string%3F%20%5B%3Afn%20%7B%3Aerror%2Fmessage%20%22should%20not%20be%20empty%22%7D%20(complement%20str%2Fblank%3F)%5D%5D#2019-10-3014:07Toni VanhalaThere’s not-empty in core, while blank? is in clojure.string#2019-10-3014:08Toni VanhalaI would expect [:and string? not-empty] to work out-of-the-box, while blank? is arguable#2019-10-3014:10ikitomminot-empty is not registered by default, should it be?#2019-10-3014:11ikitommi[:and string? [:fn {:error/message "should not be empty"} 'not-empty]]
#2019-10-3014:11ikitommiworks#2019-10-3014:11ikitommihttps://malli.io/?value=%221%22&schema=%5B%3Aand%0A%20string%3F%0A%20%5B%3Afn%20%7B%3Aerror%2Fmessage%20%22should%20not%20be%20empty%22%7D%20not-empty%5D%5D#2019-10-3014:15eskosWould be nice 🙂#2019-10-3014:16eskosAlthough I think complements are also useful in general…#2019-10-3109:47orestisI'm evaluating various "spec" libraries for use in production. It's a bit of a sticky situation, since clojure.spec is alpha, clojure.spec2 and malli are pre-alpha. The only "stable" library is Plumatic Schema, but that's also old (though actively maintained).#2019-10-3109:48orestisBut, everyone's definition of pre-alpha is different. Alex Miller explicitly says "don't use spec2 in production yet, eta is in the order of months".#2019-10-3109:48orestisWhat is the definition of pre-alpha for Malli?#2019-10-3109:57ikitommiwe wanted to optimize the whole - currently all non-core modules (generators, providers, transformers, etc) have initial versions and we know how to core should be done to support the needs of the modules. There will be few namespace & function renames (e.g. breaking) after which we’ll put a alpha out. Will be in alpha as long all the relevant features are feature-complete, but I would guess there will be a real release within 1-2 months.#2019-10-3109:59ikitommithere is for example now m/transform function but it will be split into m/encode and m/decode, like in spec-tools. Didn’t want to start maintaining CHANGELOG for all the things that are in a flux now, but final design starts to emerge, so getting close.#2019-10-3109:59ikitommialso, there are some big things that need to be decided before release, will write a post &/ poll out of those.#2019-10-3110:00ikitommi• open or closed maps by default? (open)
• support clojure core predicates (`ìnt?`, string?) by default or make keyword type schemas (`:int`, :string, ...)#2019-10-3110:02ikitommiwe’ll push a reitit coercion module of malli soon, so people can start playing (and reporting issues) with it.#2019-10-3113:18orestisThanks for the update! Looking forward to see what's cooking.#2019-11-0107:34ikitommistill surprised how well the schema/hiccup syntax supports elegant extensions: https://github.com/metosin/malli/issues/105#2019-11-0310:06eskosI'd claim that's the benefit of supple systems 🙂 Not only hiccup syntax has low mental load, it is easy to reason without switching contexts just for the specific task and being just data it can be rustled as lightly or heavily as one needs to... TBH this is my primary reason I'm now lurking here, using hiccup syntax is definitely if not the absolutely correct, at least a very good idea as it is very approachable.#2019-11-0218:10roklenarcicI have a question related to malli, but it's probably just an error on my part but I cannot spot it:
I defined macro as this (simplified example):
(defmacro x [y]
`(println ~(m/schema [:tuple y])))
And when I call (x int?) I get:
Syntax error compiling fn* at (/private/var/folders/fm/5mzhclpd7mj0tzjjq796ftc00000gn/T/form-init14721255528686671623.clj:1:1).
Can't embed object in code, maybe print-dup not defined:
However if I define macro as this it works:
(defmacro x [y]
`(println (m/schema [:tuple ~y])))
=> #'clj-rest-client.core/x
(x int?)
[:tuple int?]
#2019-11-0218:11roklenarcicI wanted to compute the schema value at macro expansion time to make it faster, but it doesn't seem to work for some reason#2019-11-0218:11roklenarcicmaybe someone else can spot it, I've been looking at this for a while with no idea#2019-11-0218:38ikitommi@roklenarcic macros should return source code, the first example returns a reified protocol instance. You can run (macroexpand ...) to that#2019-11-0218:39ikitommiJust curious, "to make it faster"?#2019-11-0218:39roklenarcicI thought that m/schema does some preprocessing#2019-11-0218:39roklenarcicwhich would make it faster than sticking simply [:tuple x] in there#2019-11-0218:40roklenarcicbtw, it seems that [:tuple] doesn;'t work#2019-11-0218:40roklenarcic(m/schema [:tuple]) -> :tuple but (m/schema [:tuple int?]) -> [:tuple int?]#2019-11-0218:41ikitommimalli has it's own compiler for validation, explain & transform. Reaults of those should be pretty fast.#2019-11-0218:42ikitommiCreation of schema has some overhead, but if you can store a reference to a transformer, it should be as fast as one can do with Clojure.#2019-11-0218:43roklenarcicthanks for the info#2019-11-0218:46ikitommithe :tuple thing... some schemas has a constraint that they should have 1+ childs. I guess tuple doesn't and empty tuple is ok...#2019-11-0218:46roklenarcicI had spec1 code that said -> give me a vector of values and a vector of specs and I will generate a s/fdef spec (which uses s/cat), so the code generated a spec like this (s/cat :arg1 arg-spec1 ...) and potentially (s/cat) which took an empty sequence just fine. I replaced this with :tuple from malli and it seems to have a corner case for empty tuple#2019-11-0218:47ikitommishould it require at least 1 child?#2019-11-0218:47roklenarcicit works the opposite#2019-11-0218:48roklenarcic(m/explain [:tuple] [])
Execution error (NullPointerException) at malli.core/-tuple-schema$reify$reify$fn (core.cljc:447).
null
#2019-11-0218:48roklenarcicbut spec with empty (s/cat) would work fine#2019-11-0218:48ikitommiOh, that's a bug.#2019-11-0218:48roklenarcicyeah I thought so#2019-11-0218:49roklenarcicNPEs are usually not intended#2019-11-0218:49roklenarcicI'm porting my rest client lib from spec to malli now... hopefully I'll have encode/decode soon#2019-11-0218:53roklenarcicI'd like for user to be able to specify the transformation to string I should do before sending any opaque types over the wire#2019-11-0218:53roklenarcictransforms seem to be doing the opposite direction#2019-11-0218:55roklenarcice.g. if user has LocalDateTime object I'd like for them to specify format in schema that I format it to before I send it...#2019-11-0219:01ikitommihttps://github.com/metosin/malli/commit/c2f6b45bd12751f18b737cb227d17fae7f024663#2019-11-0219:04ikitommiencode & decode is done (#99), but requires (#98) before will merge that in.#2019-11-0219:16roklenarcicI'll be finally rid of the nightmare that is trying to conform s/merge specs#2019-11-0300:44roklenarcicBase Registry includes schemas :vector, :list, :set, but what I'm missing here is :sequential. Often I have use cases where I don't particularly care which collection it is as long as I can process it like a sequence.#2019-11-0311:50roklenarcicHow would I write (s/coll-of string?) in malli?#2019-11-0313:01ikitommiI think :sequential would be nice. Is :list actually useful at all?#2019-11-0313:18roklenarcicI mean it is if you want to schema a list but thats so rare except when you’re validating macro arguments#2019-11-0313:19roklenarcicThe operators i miss the most are coll-of and every#2019-11-0412:50ikitommi@roklenarcic in latest master, you can use IntoSchema values as element in the schema ast:
(require '[malli.core :as m])
(def sequential (#'m/-collection-schema `sequential sequential? seq nil))
(m/validate [sequential int?] '(1 2 3))
; => true
(m/validate [sequential int?] [1 2 3])
; => true
(m/form [sequential int?])
; => [user/sequential int?]
#2019-11-0412:54ikitommihttps://github.com/metosin/malli/commit/adc9994043977965cdca3e37bfa3187c36cc6b1b#2019-11-0414:57roklenarcicnice trick to get around private var 🙂#2019-11-0417:24ikitomminot complete, transforming needs to retain the original type, but an example of all the extensions implemented and tested for a schema: https://github.com/metosin/malli/pull/110#2019-11-0417:25ikitommiI'm thinking of adding a supporting protocol for the different extension, so one could just implement all the concerns with one reify.#2019-11-0417:26ikitommimalli.json-schema/JsonSchema etc.#2019-11-0417:31ikitomminot going to use the protocols in the core, e.g. core doesn't depend on anything to keep it small, but for custom new client schemas, I think the all-protocols approach is more declarative than the mixture of protocols and mms#2019-11-0511:38plexusSeems there's an issue when using a custom registry with generate. The generators don't pass the opts down into m/schema, so when you have nested schemas they end up using the default registry#2019-11-0511:42plexusor I'm holding it wrong 🙂 trying to come up with a minimal repro#2019-11-0512:10ikitommihmm… quick look says the sequential schemas are not passing the opts. PR welcome @plexus#2019-11-0512:11plexushttps://github.com/metosin/malli/pull/112#2019-11-0512:12ikitommithanks, merged#2019-11-0512:12plexussweet!#2019-11-0512:13plexusside note, is it too late to change childs to children?#2019-11-0512:13plexusit bothers me every time 🙂#2019-11-0512:13ikitommi* children - 8 700 000 000 hits
* childs - 75 200 000 hits#2019-11-0512:14ikitommiit’s pre-alpha, good time for fixing things like this. Do you have time for a PR?#2019-11-0512:15ikitommi> The difficulty with the plural began in Old English, where the nominative plural was at first cild, identical with the singular, then c.975 a plural form cildru (genitive cildra) arose, probably for clarity’s sake, only to be re-pluraled late 12c. as children, which is thus a double plural. Middle English plural cildre survives in Lancashire dialect childer and in Childermas.#2019-11-0512:15plexusinteresting! wiktionary is a bit more harsh#2019-11-0512:16plexus#2019-11-0512:16plexusyeah, I can do a PR for that later today#2019-11-0512:30plexusI see I also broke some tests... will send a PR for that as well#2019-11-0513:45plexusnew PR sent. I really appreciate that the project has a bin/kaocha, it's so great not to have to think about how to run the tests on each project.#2019-11-0514:00ikitommithanks for the PR kaocha, makes things kinda simple 🙂#2019-11-1217:15ikitommitwo new PRs in master: support for two-way transformations (`m/decode` & m/encode), :sequential schema and allowing any schema properties based (serializable!) encoders & decoders#2019-11-1217:21ikitommi(let [schema (-> [:and {:title "lower-upper-string"
:decode/string '(constantly str/upper-case)
:encode/string '(constantly str/lower-case)}
string?]
(malli.edn/write-string)
(malli.edn/read-string))]
(as-> "kikka" $
(m/decode schema $ mt/string-transformer)
(doto $ prn)
(m/encode schema $ mt/string-transformer)
(doto $ prn)
schema))
; prints "KIKKA"
; prints "kikka"
; => [:and
; {:title "lower-upper-string"
; :decode/string (constantly str/upper-case)
; :encode/string (constantly str/lower-case)}
; string?]
#2019-11-1217:26ikitommiThe two-way thing looks from outside the same as with spec-tools, but it actually works 🙂#2019-11-1217:27ikitommi… thanks to clear separation of transformation and validation. the encode mostly creates values that are not valid (e.g. a string representation of a date), and had to add all kind of hacks to make that work with clojure.spec.#2019-11-1406:20eskosI’m not sure if I like that str is directly usable in schemas like that. From user point of view I’d assume my own namespace requires to apply, not implicit ones from within malli and in that case I’d rather see an error…I do see the usefulness of that, but it makes me go a bit :face_with_raised_eyebrow: Eh, maybe that’s just me…#2019-11-1406:45ikitommiyou can explicitly allow certain qualified symbols to be used, could allow that via global options. I guess clojure.string is a dependency and is available for users of malli. BUT, might fail with advanced compilation unless explixitely registered those functions. Explicit is always better…#2019-11-1406:48ikitommithough of exposing explicitely some clojure.test.check.generators functions for generators, to enable portable generators with :gen/gen schema property. Those definitions should be in optional ns so if not needed, doesn’t make the bundle size bigger.#2019-11-1406:49ikitommibetter portable example in a tweet: https://twitter.com/ikitommi/status/1194555187995848705#2019-11-1406:51ikitommi@suomi.esko actually, the str stuff is explicitly allowed in sci, so ok: https://github.com/borkdude/sci/blob/9b70708c5c42e89d94a72aa961bfa1d3a6ef8be4/src/sci/impl/namespaces.cljc#L437-L475#2019-11-1406:52eskos¯\(ツ)/¯#2019-11-1518:34pithylessWhat is the expected way to check for a function? Something like [:map [:handler fn?]]
It looks like fn? is not one of the symbols exported in https://github.com/borkdude/sci/blob/cb1dc3139a53c680b2bdc1038d2c6e26b975ee8e/src/sci/impl/namespaces.cljc#L96
I could do something like [:map [:handler [:fn fn?]]], but I'm wondering if this is something that will be supported for serialization.#2019-11-1519:24ikitommiI think @borkdude knows if sci could support fn?#2019-11-1519:25borkdudesure. PR welcome, it should be added about here: https://github.com/borkdude/sci/blob/ab5ba82fa61ab47b7ace8b3e58bfa633ae8e1f72/src/sci/impl/namespaces.cljc#L201#2019-11-1519:25borkdudeslipped through, don't know why#2019-11-1519:26ikitommiAwesome, [:map [:handler [:fn 'fn?]]] would work after that & survive the serialization#2019-11-1622:32borkdudeI just added fn? to sci master#2019-11-1806:13danielgrosseHi, when adding the latest ref from malli in the deps.edn and starting a repl, I've got this error: error in process sentinel: Could not start nREPL server: Error building classpath. Unable to compare versions for borkdude/edamame: {:mvn/version "0.0.8-alpha.4", :deps/manifest :mvn} and {:git/url "", :sha "34755e7a875fae2ab4d64e6be89b6270046a91a3", :deps/manifest :deps, :deps/root "/Users/xxx/.gitlibs/libs/borkdude/edamame/34755e7a875fae2ab4d64e6be89b6270046a91a3"}
#2019-11-1806:14danielgrosseAny tips how I could solve this?#2019-11-1806:21ikitommi@danielgrosse fixed that on master - for edamame, it uses now the “0.0.8-alpha.4” version. DEPS doesn’t know which one is latest that or the SHA (which was later).#2019-11-1806:21ikitommi➜ ~ clj -Sdeps '{:deps {metosin/malli {:git/url ""
:sha "e2ff736ff923c676de655b853180be2503715d88"}}}'
Checking out: at e2ff736ff923c676de655b853180be2503715d88
Clojure 1.10.1
user=>
#2019-11-1806:22ikitommiNot sure if that is a transitive dependency bug of deps, or just a feature.#2019-11-1806:23ikitommi@pithyless master uses the latest version of sci, so [:map [:handler [:fn 'fn?]]] should work now.#2019-11-1806:24danielgrosseThanks for the fast fix. :thumbsup:#2019-11-1806:31ikitommiIt was an easy fix 😉#2019-11-1806:32ikitommibtw my next steps are to finish :multi schema and the pretty error printer btw, both 80% done.#2019-11-1806:32danielgrosseCool. Keep up the great work.#2019-11-1806:33ikitommirecursive schemas & support for local registries might come via this PR https://github.com/metosin/malli/pull/117#issuecomment-554737541#2019-11-1809:40eskosTangential thought on documentation, would absolutely love to see documentation like this for malli eventually: https://pomb.us/build-your-own-react/ scroll a bit to see what I mean - the library behind that is https://github.com/pomber/code-surfer/blob/code-surfer-v2/readme.md#2019-11-1812:53ikitommi@suomi.esko feel free to build the docs. looks good, despite not being interactive.#2019-11-1917:23eskosI’ll try to give this a swing when I get some free time just to see if the idea actually works.#2019-11-1917:23eskosGives me a good excuse to really dive into malli as well 🙂#2019-11-1918:10ikitommilooking forward to seeing the new docs ;)#2019-11-2006:58eskosthey might end up being trash ¯\(ツ)/¯ but then we’ll know why#2019-11-1914:00ikitommiOh, the :multi popped out from the train: https://github.com/metosin/malli/pull/118#2019-11-2010:28kszaboAre there any plans of supporting clojure.spec1/2? The value proposition of Malli looks nice, but tooling has moved towards supporting clojure.spec and to ease adoption an iterative approach would be nice, slowly redefining existing system clojure.specs to Malli based data-oriented schemas.#2019-11-2010:45miikkaWhat would supporting spec mean?#2019-11-2010:47miikkaI suppose you could try to compile Malli schemata to spec-tools data specs#2019-11-2010:47miikkaIn my experience, mixing clojure.spec and prismatic schema was not a big deal, though, so not sure there's big need for integration in practice.#2019-11-2010:51Toni Vanhala> I suppose you could try to compile Malli schemata to spec-tools data specs
I think the requirement was other way around “redefining existing system clojure.specs to Malli”#2019-11-2010:57ikitommino activities currently for the interop with spec. Feel free to suggest new tooling for this.#2019-11-2011:33kszaboactually both directions would be great 🙂 thanks for the notice#2019-11-2106:17ikitommiAs schemas are responsible for parsing their own ast, we could have container schemas for doing translation between modelling systems. Doesn’t help with spec as it’s built around the global registry & macros, but would work with declarative models like JSON Schema:
[:json-schema
{:type "object"
:properties {:name {:type "string"}
:number {:type "number"}
:required [:name]}]#2019-11-2211:13roklenarcicNot having a global registry has a downside though. Let’s say some library introduces a new Schema protocol implementation. Then someone uses that in schemas in namespace X. With Spec when you load the namespace you automatically gain all specs in there with all the required bits and pieces to make it all work. But with the way Malli is set up, if something is using a custom :datetime schema, then all the users of that thing must invoke all the Malli functions with an opts map where :registry is set to the default one with all the extensions added. If the calls to m/validate and such are burried inside a third party library, then that library must provide a way for you to pass the registry to be used otherwise, you cannot use any new schema types except the ones defined in malli.core#2019-11-2211:14roklenarcicAlso I’d like to ask, what is the policy regarding non-serializable content in schemas. I see that some examples include opts map where you have keys like {:decode/string some-fn} which isn’t really serializable#2019-11-2211:23roklenarcicThis is meant vis-a-vis developing own schema implementations… should I disallow non-serializable options?#2019-11-2212:09ikitommi@roklenarcic you can always define the library schemas as Vars, the first value in the Schema vector syntax can be an instance of IntoSchema.#2019-11-2212:10ikitommi(def date-time (reify IntoSchema ...))
(m/validate date-time (java.time.DateTime.))
; => true
#2019-11-2212:11ikitommithe work like components on Reagent.#2019-11-2212:13roklenarcicso I can also do `[datetime “format”]?#2019-11-2212:13ikitommipolicy on non-serializable: up to the user. There is an issue about making an utility that checks if a schemas can be fully persisted or not. One can put that into project tests.#2019-11-2212:13ikitommiyes#2019-11-2212:13roklenarcicthanks that is very helpful#2019-11-2212:13roklenarcicI mean any schema that specifies decode/encode implementation cannot be persisted?#2019-11-2212:13ikitommi(defprotocol IntoSchema
(-into-schema [this properties children opts] "creates a new schema instance"))
#2019-11-2212:14ikitommiquote the functions and they can be serialized.#2019-11-2212:14ikitommi(require '[malli.core :as m])
(require '[malli.edn :as edn])
(-> [:and
[:map
[:x int?]
[:y int?]]
[:fn '(fn [{:keys [x y]}] (> x y))]]
(edn/write-string)
(doto prn) ; => "[:and [:map [:x int?] [:y int?]] [:fn (fn [{:keys [x y]}] (> x y))]]"
(edn/read-string)
(doto (-> (m/validate {:x 0, :y 1}) prn)) ; => false
(doto (-> (m/validate {:x 2, :y 1}) prn))) ; => true
;[:and
; [:map
; [:x int?]
; [:y int?]]
; [:fn (fn [{:keys [x y]}] (> x y))]]#2019-11-2212:14roklenarcicah.. but then the code processing the value has to call eval or something on it?#2019-11-2212:15ikitommiit uses sci , a small interpreter. So it works on JVM, ClojureScript & GraalVM, no eval needed#2019-11-2212:15ikitommiit’s <10kb on cljs.#2019-11-2212:16ikitommihttps://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D#2019-11-2212:16ikitommithat’s all normal code, no clojurescript compiler used in the demo.#2019-11-2212:17roklenarcicsorry, I should be clearer. If my custom datetime schema has an option :formatter which someone puts the value of DateTimeFormatter/ISO_INSTANT#2019-11-2212:17ikitommie.g. “production code”#2019-11-2212:17ikitommioh, that. can’t serialize that.#2019-11-2212:17roklenarcicif I quote this, I get a symbol#2019-11-2212:17ikitommiit could have :format option, which could be string?#2019-11-2212:18borkdudethe latest version of sci supports Java reflection, if you bring this class in with the :classes option.
but that won't work in CLJS of course#2019-11-2212:18roklenarcicsure, but you cannot specify instant formatter by giving a string#2019-11-2212:18borkdudebut you can use reader conditionals for that maybe#2019-11-2212:18roklenarcicspecifying something like “yyyy-MM-dd” then telling it to format an Instant object will fail#2019-11-2212:19ikitommi{:format :iso_instant} maybe?#2019-11-2212:19ikitommitime is hard to get do in a portable way anyway.#2019-11-2212:19roklenarcicMy current idea is to support everything#2019-11-2212:19roklenarcicso if someone puts a formatter instance there, then I will use it#2019-11-2212:20ikitommistring or a (predefined) keyword?#2019-11-2212:20ikitommi.. or a class, which is used.#2019-11-2212:20roklenarcicbut then they lose serializability, but that’s on them#2019-11-2212:20roklenarcicok… this was very helpful#2019-11-2212:21ikitommione thing there could be is to have a startup-time altering of a registry. via JVM options for example. It would be explicit, but still global.#2019-11-2212:22ikitommiplan is to pull schema defn and fn syntax helpers, can’t pass easily any custom options/registry there.#2019-11-2212:23ikitommi(m/defn plus :- int?
[x :- int?, y :- int?]
(+ x y))
#2019-11-2212:25ikitommito support something like [:json-schema {:type "string"}] schema would require some way to change the one registry. could just be the var reference (`[json/json-schema {:type "string"}]`), but could be a JVM property to bootstrap some extra schemas into the registry… not sure if thi is a good idea#2019-11-2212:30ikitommi… or just malli.evil ns, which has all the public functions of other namespaces, but with the default-registry as an atom.
(require '[malli.evil :as evil])
(evil/register! ::int int?)
(evil/validate ::int 1)
; => true
#2019-11-2212:31ikitommi… or a JVM option to define the default-registry implementation. by default, it points to the immutable map, but could be swapped to atom impl. Bad Idea 🙂#2019-11-2414:21ikitommiabout closed/open schemas, comments welcome on this: https://github.com/metosin/malli/issues/31#issuecomment-557892061#2019-11-2417:33rschmuklerHey @ikitommi thanks for your response on https://github.com/metosin/malli/pull/119#2019-11-2417:34rschmuklerThis feature is currently blocking me, and I'm relatively familiar with malli's code at this point (lots of tracing to figure out how to implement the above - if you had ideas on what you were thinking in terms of implementation for #120 I could potentially submit a PR#2019-11-2417:34rschmuklerOne possible solution that comes to mind is to just extend the Schema protocol to have a -parent method and then invoke things as appropriate#2019-11-2417:35rschmuklerAlso, regarding the public m/parent function, perhaps we could also specify passing in a name to traverse up parents until one with the provided name is found#2019-11-2417:35rschmuklereg (m/parent schema :map)#2019-11-2417:44ikitommi@rschmukler I think the -parent is the way to go, but it might need some bigger refactoring on internals: when a Schema is created, the -father is not yet available, as we are creating the childs from the mothers “constructor”, so it needs to be a Delay or similar value. Derefon that while creating a child would either return nil or block indefinitely, depending on the implementation. So, would have to see the code after the change to know if that’s a good value for adding the (potential) complexity in.#2019-11-2417:44ikitommiif that is blocking you, you could add a custom decoder on the :map schema already?#2019-11-2417:45rschmukler@ikitommi I initially went down that path but then I had to do the key tracking to invoke the child schemas appropriately#2019-11-2417:45rschmuklerI was actually wondering if we might be better of implementing the coercions as a postwalk instead of a prewalk#2019-11-2417:46ikitommi(m/decode
[:map {:decode/my-thing (fn [schema opts] ...do...thing...)} [:a int?] [:b int?]]
{"a_kikka" 12, "b_kikka" 32}
(mt/transformer {:name :my-thing))#2019-11-2417:46rschmuklerThe issue becomes that if the map renames a key, the child schemas don't get mapped#2019-11-2417:47rschmukler(because the map now has a new key at a different path, and the transformers for the values are resolved via key, after the key is renamed#2019-11-2417:48rschmukler(Hence why I potentially suggest doing a postwalk instead of a prewalk for coercions) - is there any reason that it's currently implemented as a prewalk?#2019-11-2417:52rschmukler@ikitommi I have another PR that I'm going to submit that might be a simpler solution for what I'm trying to achieve - feel free to reject it#2019-11-2418:22ikitommiIt has to be prewalk, good example:
(m/decode
[:multi {:dispatch :type
:decode/string '(constantly #(update % :type keyword))}
[:sized [:map [:type [:= :sized] [:size int?]]]
[:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
{:type "human"
:name "Tiina"
:age "98"
:address {:country "finland"
:street "this is an extra key"}}
(mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
;{:type :human
; :name "Tiina"
; :address {:country :finland}}
#2019-11-2418:23ikitommithere will be two-way transformers using interceptos, one could put all the stuff into :leave to get postwalk#2019-11-2418:24rschmukler@ikitommi Interceptors is a great idea#2019-11-2418:35rschmukler@ikitommi are you open to applying the map transformer after the key and value transformer for the :map schema?#2019-11-2418:36rschmuklerie. Ideally I'd like to be able to have:
(is (= {:x_key "true", :y 1} (m/decode schema {:x true :y "1"}
(transform/transformer
{:name :custom
:decoders
{:map
(constantly (fn [map]
(if-not (contains? map :x)
map
(-> map
(assoc :x_key (:x map))
(dissoc :x)))))
'boolean? (constantly str)}}))))#2019-11-2418:43rschmuklerActually this feels inconsistent - perhaps just implementing the interceptors is the way to go#2019-11-2420:13ikitommi@rschmukler what is your schema in the example?#2019-11-2420:15rschmukler[:map [:x boolean? :y int?]]#2019-11-2420:15rschmukler(although lets assume that I composed with transformer/json-transformer so that the number decodes correctly#2019-11-2420:16rschmuklerI realize that I can always pop an escape hatch and write a :custom/decode function but since I want to apply it on every map, it seems like a transformer is the idiomatic way to go#2019-11-2420:17rschmuklerI think interceptors solve everything so I'm trying my hand at implementing them right now#2019-11-2420:17ikitommibut, isn’t that m/encode you are doing?#2019-11-2420:18ikitommidecode should end up in valid values against the schema (eg. JSON->Schema), encode is Schema->JSON.#2019-11-2420:19rschmuklerThat's a buggy example haha#2019-11-2420:19rschmukler(mine above)#2019-11-2420:19rschmuklerIt is indeed an encode#2019-11-2420:20rschmuklerBut, none the less, I believe after the key is renamed during the encode, the subsequent value transformers will fail because they look up by the original key name#2019-11-2420:21rschmukler(ie. (if-let [entry (find m k)] (assoc m k (t (val entry))) I believe is the code in the map schema)#2019-11-2420:22ikitommitrue that. interceptors are needed#2019-11-2420:22rschmuklerYeah, hoping to have a PR today#2019-11-2420:22rschmukler😄#2019-11-2420:23ikitommiof interceptors? that’s a big one.#2019-11-2420:23rschmuklerRight now I'm having it so that -transformer and -value-transformer return {:enter _ :exit _} maps#2019-11-2420:24rschmuklerand then I'm updating the schema impls to use those on children as needed#2019-11-2420:24ikitommithere is https://github.com/metosin/malli/issues/114#2019-11-2420:24rschmukler(defn- ->interceptor
"Utility function to convert a transformer into an interceptor. Works with transformers
that are already interceptors, as well as sequences of transformers"
[transformer]
(cond
(fn? transformer) {:enter transformer :exit nil}
(and (map? transformer)
(or (contains? transformer :enter)
(contains? transformer :exit))) transformer
(coll? transformer) (reduce
(fn [{:keys [enter exit]} {new-enter :enter new-exit :exit}]
(let [enter (if (and enter new-enter)
(comp enter new-enter)
(or enter new-enter))
exit (if (and exit new-exit)
(comp exit new-exit)
(or exit new-exit))]
{:enter enter :exit exit}))
(map ->interceptor transformer))
(nil? transformer) nil
:else (throw (ex-info "Invalid transformer. Must be a function, sequence, or interceptor map"
{:value transformer}))))
#2019-11-2420:25rschmuklerI think that solves it, converting all transformers into an interceptor, even allowing for the vectors that you included#2019-11-2420:25rschmukler(in 114)#2019-11-2420:25rschmuklerthat map call should be a keep#2019-11-2420:26rschmuklerThen, like I said, basically all Schema impls need to be aware of :enter and :exit when returning their own interceptor (with :enter and :exit) themselves#2019-11-2420:26ikitommiawesome, looking forward to the PR. (`:exit`=> :leave would be coherent with interceptor libs)#2019-11-2420:26rschmuklerand then, I think encoder / decoder / encode / decode just need to call (comp exit enter)#2019-11-2420:26rschmuklerAh, yes, will rename the key, thank you!#2019-11-2420:26ikitommi👍#2019-11-2420:26rschmuklerDo you want me to rename the -transformer and -value-transformer fns to -interceptor and -value-interceptor?#2019-11-2420:27rschmukler(no opinions here)#2019-11-2514:15roklenarcicso what happens when you use mt/transformer to combine two existing transformers that have an encoder/decoder for same schema name?#2019-11-2514:17roklenarcicseems to me that the last encoder/decoder function wins#2019-11-2514:18roklenarcicso if you have a modifying transformer like strip-extra-keys-transformer and someone else defines a transformer for :map then one of them doesn’t work, if I am not mistaken#2019-11-2514:18ikitommishould put into chain (after interceptors)#2019-11-2514:19ikitommiI think#2019-11-2514:20roklenarcicchain?#2019-11-2514:20roklenarcicis this some new branch?#2019-11-2514:22roklenarcicshould I try to tamper with -transformer function to achieve a modifying effect#2019-11-2516:01rschmukler@roklenarcic https://github.com/metosin/malli/pull/122#2019-11-2516:02rschmuklerIt'd be reasonably easy to move the ->interceptor function in to have it compose if there's already existing transformers#2019-11-2516:03rschmukler@ikitommi it might be worth having it compose with both :enter and :leave steps#2019-11-2516:03roklenarcicyou have a comment block at the top there#2019-11-2516:03rschmuklerI just noticed that! Thanks!#2019-11-2516:05rschmuklerhttps://github.com/metosin/malli/pull/122/files#diff-f4026c83e07e6f542344d5945d310607R12#2019-11-2516:06rschmuklerSo basically on the transformer call where it calls ->interceptor - you could move that reduce internally and then later transformers in & options could just compose#2019-11-2516:08roklenarcicmaybe schema seems to be reporting wrong name in error message#2019-11-2516:08roklenarcichttps://github.com/metosin/malli/blob/master/src/malli/core.cljc#L556#2019-11-2516:17ikitommi.. about composing the transformers. What should happen when there are multiple transformers for same key?
1. last one wins (current)
2. all transformers with different name get chained
3. all get chained#2019-11-2516:18ikitommi(mt/transformer mt/json-transformer mt/json-transformer)
1. normal json
2. normal json
3. json + json#2019-11-2516:21ikitommi(mt/transformer {:decoders {:map keywordize-keys}} mt/strip-extra-keys-transformer)
1. strip-keys wins
2. both
3. both#2019-11-2516:22rschmukler@ikitommi I wouldn't do unique based off of name - I think for people unfamiliar with the implementation, you might not think that name matters much. Someone might write their own :json that they also compose with the built in json#2019-11-2516:22rschmuklerWe could do all unique functions though#2019-11-2516:22ikitommi(mt/transformer {:decoders {:map keywordize-keys}} {:decoders {:map upper-case-keys})
1. upper-case wins
2. upper-case wins
3. both#2019-11-2516:23rschmuklerI think both would be most useful#2019-11-2516:23rschmuklerthen you can write things like (mt/transformer keywords-as-strings capitalize-keywords)#2019-11-2516:24rschmukler(I could even see utility libraries existing with tons of tiny compositions of common transformer needs)#2019-11-2516:25rschmuklerAlso, I actually take back my comment on using unique functions#2019-11-2516:25rschmuklerI'd say let the user compose things however they want#2019-11-2516:36rschmukler@ikitommi if you'd like I can add that functionality to my interceptor PR#2019-11-2517:04eskosI almost commented on GitHub but not sure where this thought would actually fit, I started pondering that what if the data given to interceptor would be a zipper like navigation cursor instead? This would actually allow the interceptor to walk up/down/left/right the data structure when needed, without "bind to parent to get access" kind of tricks.#2019-11-2517:37rschmuklerPersonally I prefer the interceptor pattern as a balance of simplicity, power, and because it's common in other libraries in the ecosystem. The cursor is definitely much much much more powerful, but building it will be non-trivial. Specifically, the way schema expansion currently happens is sort of a pre-walk - so a child doesn't even have reference to its parent because when the child is instantiated the parent doesn't even exist yet.#2019-11-2517:42rschmuklerThe other thing of note is that right now all of the traversal / introspection of the schema happens at compile time (or, at encoder / decoder ) time. I'm not familiar enough with zippers to think through how mapping the functions produced by the zipper could then be composed into a transformer#2019-11-2517:55eskosI'm almost certain it would be runtime only and probably not that incredibly fast :)#2019-11-2518:00eskosBut I disagree on that sentiment of simplicity, as I don't see it as simple for any other than the simple use case - this parent awareness being a very good example of a sudden trickery situation, esp. with multi schemas where the parent doesn't necessarily dictate which child there is.
It is always possible everyone's gotten things wrong so far... 🙂#2019-11-2518:01rschmuklerIt absolutely is!#2019-11-2518:01rschmuklerI agree with you 100%#2019-11-2518:02rschmuklerBecause the schema's are data, we at least have the escape hatch of compiling a different schema#2019-11-2518:03rschmuklerBut that feels "dirty" to me - not sure why... maybe bad habits from spec#2019-11-2518:04eskosI've never used spec beyond tutorials tbh, I disliked its global registry and nested specs too much.#2019-11-2518:04eskosSo maybe I'm not tainted/enlightened in that sense? 🙂#2019-11-2518:04rschmuklerLucky you 😉#2019-11-2518:05eskosBut anyway, it definitely is a concern that it would probably be really slow (in relation to rest of malli)#2019-11-2518:05rschmuklerThe tooling story for spec is pretty cool. Definitely something I intend to port over to malli once things stabilize a bit#2019-11-2518:05rschmukler(>defn- wrap-vec
"Wraps the given target into a vector. If its already in a sequence, it will
be converted to a vector. If its a vector it will return itself. If its a value
it will be inserted into a vector"
[item]
[any? => (s/coll-of any? :kind vector?)]
(cond
(vector? item) item
(seq? item) (into [] item)
:else [item]))
#2019-11-2518:05rschmuklerThat kind of inline annotation + optional checks is pretty useful#2019-11-2518:06rschmukleroptional instrumentation I should say#2019-11-2520:48ikitommiI would prefer the plumatic schema fn syntax:
(m/defn plus :- int?
[x :- int?, y :- int?]
(+ x y))
#2019-11-2520:49ikitommieffectively about the same.#2019-11-2522:17rschmukler@ikitommi interesting, why the preference? You like the type annotations right next to the variable name?#2019-11-2522:32rschmuklerI can likely make both valid. I'll probably just have different namespaces that you can import from#2019-11-2615:57ikitommi@rschmukler been using that with schema for 5+ years, no complaints. Cursive does static analysis for that + you can omit the type hints if you don't know/want to define those, e.g
(m/defn plus
[x :- int?, y]
(+ x y))
#2019-11-2615:58ikitommiwill check the interceptor PR as soon as have extra time.#2019-11-2615:59ikitommithe chaining of encoders & decoders with mt/transformer, I think it's a good idea.#2019-11-2616:13ikitommihttps://github.com/danielcompton/defn-spec#2019-11-2617:02kszabothere is also: https://github.com/fulcrologic/guardrails/#2019-11-2617:06ikitommiand https://github.com/jeaye/orchestra/blob/master/README.md#defn-spec#2019-11-2617:07ikitommiIs there something missing from the original plumatic schema syntax?#2019-11-2619:26rschmukler@ikitommi I’ve got a proof of concept. I’ve separated the parsing from the code generation (the generation just takes a map). This will allow different styles to work.#2019-11-2619:28rschmuklerRegarding the composition of interceptors, if you don’t mind, I may do it as a separate PR to keep thins more focused.#2019-11-2705:39eskosMy only gripe with Schema’s defn macros is that there’s no private versions and IIRC they sort of b0rked if you try to include type information or other metadata…but those are all fixable issues 🙂 And I might be misremembering things, I’m quite good at that.#2019-11-2807:40eskos@ikitommi As morning chore I made a Makefile for Malli 😛 mainly meant to replace the bin/kaocha and to make sure deps are installed. Since no one even asked for this, should I push this out or just keep to myself?#2019-11-2906:02eskos@ikitommi So I made a PR of this anyway, feel free to reject it if you disagree with my reasoning :)#2019-11-2906:10ikitommiThere are smarter on Metosin like @U1NDXLDUG doing our OS tooling stuff, looking forward to comments too.#2019-11-2906:31miikkabin/kaocha should not be replaced, it’s essential part of the Kaocha interface#2019-11-2906:33eskosPlease elaborate 🙂#2019-11-2906:33eskosI’m replacing the literal bash file, not kaocha itself.#2019-11-2906:34miikkaYes, but Kaocha is supposed to be used from command-line#2019-11-2906:34miikkaYou can do stuff like bin/kaocha --watch and bin/kaocha --focus test-symbol and stuff like that#2019-11-2906:34miikkaThat said, I wouldn't be against something like make test which would install the npm deps and then run the tests with bin/kaocha#2019-11-2906:35eskosAh, I see.#2019-11-2906:35eskosWell, that’s not a big adjustment 🙂#2019-11-2906:43eskosThere. I did the unfashionable and hid my crimes 😉#2019-11-2907:58miikkaI'm honestly not feeling very positive about your PR, but I'll try to review it at some point.#2019-11-2908:55eskosThat’s totally fine 🙂 Just wanted to push it out to clarify your stance on it.#2019-11-2819:59ikitommi@rschmukler Merged the interceptor-branch, thanks! Wanted to test myself if/how that works and re-formatted the code: https://github.com/metosin/malli/pull/124/files?w=1#2019-11-2820:01ikitommi(is (= [24 48 8 10]
(m/decode
[:tuple
{:decode/string (constantly {:enter (partial mapv inc), :leave (partial mapv (partial * 2))})}
[int? {:decode/string (constantly {:enter (partial + 2), :leave (partial * 3)})}]
[int? {:decode/string (constantly {:enter (partial + 3), :leave (partial * 4)})}]]
[1 2 3 4] mt/string-transformer)))#2019-11-2820:13ikitommidid an issue about the defn syntax, labelled “for discussion”.#2019-11-2820:56rschmukler@ikitommi Woo! Looks great 🍻#2019-11-2820:57rschmuklerI was just recently thinking that there are probably some interesting behaviors we should add for multi schemas. Eg. should we have a :dispatch/leave option? and then assume :dispatch and :dispatch/enter to be the same?#2019-11-2905:51ikitommihmm… I think the :dispatch doesn’t need a separate leave?#2019-11-2905:52ikitommias the dispatch is used to select the schema when going in, when returning, we don’t need the selection, we just go back the same route.#2019-11-2906:03ikitommiHmm,.. after mt/transformer chains the interceptors for the same schema key (instead of overriding), what if the names are also chained so that one could define multiple schema-based decode/encode via schema properties?#2019-11-2906:07ikitommie.g. (mt/transformer {:name :enter} mt/string-transformer {:name :leave}) would make a name chain of [:enter :string :leave] which would allow fine grained control of overriding things in the schemas:
(m/decode
[:and {:decode/enter (constantly inc),
:decode/string (constantly (partial * 2))
:decode/leave (constantly (partial + 10))} int?]
1
(mt/transformer {:name :enter} mt/string-transformer {:name :leave}))
; => 30#2019-11-2906:08ikitommimore common case would be to “after normal decoding, do this too”:
(m/decode
[:and {:decode/leave (constantly (partial + 10))} int?]
"1"
(mt/transformer {:name :enter} mt/string-transformer {:name :leave}))
; => 11#2019-11-2906:08ikitommigetting complex, not sure if this is a good idea.#2019-11-2906:12eskosNot going to lie, I had to read that about 5 times to grok it…#2019-11-2906:13ikitommi(m/decode
[:and [int? {:decode/string '(constantly {:enter (partial + 2), :leave (partial * 3)})}]]
"1"
mt/string-transformer)#2019-11-2906:13ikitommicurrently, that doesn’t work, as there is no way to compose the normal transformation and a custom one. if you define anything at schema-level, it overrides the default. might be ok and just needs to be documented.#2019-11-2906:21ikitommibtw, do you @borkdude have a defn-parser in some of your utility libs? would be great not need to reinvent the wheel with the schema defn -syntax, expecially if malli already depends on that functionality (sci or edamame)#2019-11-2906:25eskosYou can rip it out of Clojure quite easily (https://github.com/clojure/clojure/blob/clojure-1.9.0/src/clj/clojure/core.clj#L283)#2019-11-2906:31eskosAnd if that feels dangerous, I’ve made one incomplete implementation of defn parser with clojure.tools.reader a while back because I needed to analyze defn s (don’t ask) and ended up with this sort of bleh thing which eats defn s and produces data maps.#2019-11-2906:31eskosI’m sharing that just for the idea, it’s obviously not complete 😛 In fact I should continue this project, I’ve been sitting on it for far too long…#2019-11-2907:55borkdude@ikitommi there is a defn parser in sci of course but I haven’t thought about sharing it so it might be a little specific for sci #2019-11-2907:57borkdudeIn edamame there is a fn literal -> fn parser feature #2019-12-0218:21ikitommiintegrating into reitit (wip): https://github.com/metosin/reitit/pull/341#2019-12-0218:25ikitommiWe can create all the needed functions per schema & transformer ahead of time:
(->Coercer (m/decoder schema t)
(m/encoder schema t)
(m/validator schema)
(m/explainer schema))#2019-12-0516:10rschmuklerSorry about that @ikitommi!!#2019-12-0517:38roklenarcicSequential schema doesn’t seem to be working correctly:
(m/encode [:sequential {:encode/string (constantly #(clojure.string/join "," %))}
string?]
["A" "B" "C"]
mt/string-transformer)
=> (\A \, \B \, \C)
instead of expected: "A,B,C"#2019-12-0517:39roklenarcicthe encode function correctly receives the input vector, and returns a string#2019-12-0517:39roklenarcicthe result should be a string#2019-12-0522:15rschmukler@roklenarcic I actually just ran into the same bug in a different manifestation! @ikitommi PR incoming...#2019-12-0523:05rschmukler@roklenarcic https://github.com/metosin/malli/pull/135#2019-12-0523:27roklenarcicWhat is the use case for the “interceptors” in Malli? What use cases did it solve? Because all I see is things getting more complicated and yet it adds no features. The interceptor change smells like cargo cult programming to me. Because http request/response frameworks have switched to interceptors, people think they are applicable to every problem. I just don’t see what it adds to Malli#2019-12-0523:29rschmuklerYou're right that in their current form they aren't that useful. The big draw to them would be if the :leave could execute in a more post-walk fashion. Right now, for simple use cases, they give an opportunity for a transformer to finalize a value after most things are done (ie. all :leave execute after all :enter have finished). I'm hoping that we can make it a true postwalk, in which case they become much more useful#2019-12-0609:07roklenarcicBut I was able to do that in the old system as well. My transform function was simply: call the child transformer and then finalize and return a value.#2019-12-0609:39ikitommi@roklenarcic could you show example how do you call the child transformer from a transformer?#2019-12-0609:40ikitommithe lib is at pre-alpha , so we can still rollback anything if there is unneeded complexity around#2019-12-0609:40roklenarcicAssuming you have in -into-schema schema' (-> children first (m/schema opts))#2019-12-0609:42roklenarcic(-transformer [this transformer context]
(let [child-xf (m/-value-transformer transformer schema' context)]
(if child-xf
(if (= :decode context)
(comp child-xf xf)
(comp xf child-xf))
xf)))#2019-12-0609:42ikitommiexample: you want to transform map after the keys have been renamed, using transformed keys. This is a good place to do it in leave.#2019-12-0609:44roklenarcicyou can create a transformer like this:
my enter logic
call child
my leave logic
#2019-12-0609:45roklenarcicthe difference is that in the interceptor pattern, calling the child (i.e. next in chain) is handled by the function that runs the whole thing#2019-12-0609:46roklenarcicbut in http request/response interceptors that code is doable because it is obvious how to handle calling the next interceptor in chain#2019-12-0609:46roklenarcicthere is 0 or 1 next interceptors#2019-12-0609:47roklenarcicin malli, you have maps and vectors and custom types, which means you cannot have standard function which will invoke next interceptors and combine results#2019-12-0609:53ikitommithe current interceptor runner is quite simple, but my idea was to use next version of sieppari, on which, the chain is a vector of IntoInterceptor instances, e.g. functions, maps or Interceptor Instances. The runner composes the (optimized) chain. This allows things like visual chain debugging, good perf and overall, and reuse.#2019-12-0609:54roklenarcicAre you talking about multiple interceptors on same schema?#2019-12-0609:54ikitommimanual chaining is not that handy if you want to inline the transformation via schema properties#2019-12-0609:54ikitommiyes#2019-12-0609:55roklenarcicor are you talking about composing say an interceptor on :sequential schema with those on child schemas#2019-12-0609:55ikitommihttps://github.com/metosin/malli/blob/master/test/malli/core_test.cljc#2019-12-0609:56ikitommiadded some tests to document how the chaining works now. Grep for :enter#2019-12-0609:57roklenarcicThose are serializable?#2019-12-0609:57ikitommiYes#2019-12-0609:57roklenarcic#() is serializable?#2019-12-0609:57roklenarcicI never knew#2019-12-0609:58ikitommiThanks to sci by @borkdude. Just quote them and it works#2019-12-0609:58roklenarcicas said, the biggest difference is that in request/response interceptor chains, they are strictly linear#2019-12-0609:59ikitommiMalli uses a safe/always-terminating subset of Clojure core with sci#2019-12-0609:59ikitommiis non-linear also needed?#2019-12-0610:01ikitommi#2019-12-0610:02ikitommithat's sieppari-based data visualization that we get for free with intercwptors#2019-12-0610:02roklenarcic:map-of and :vector combine enter and exit function in non-linear maner#2019-12-0610:03ikitommihmm. haven't had time to check out that. Could you please explain?#2019-12-0610:04roklenarcicyes, when writing -transformer function you would expect that with interceptors one doesn’t have to do the composing with the next interceptor in chain#2019-12-0610:05roklenarcicbut you still have to manually write -transformer function to compose your interceptor with the others#2019-12-0610:06roklenarcice.g. in :vector you need to write code that will return a map {:enter … :leave ...} such that it correctly composes its enter/leave logic with the elements enter/leave logicks#2019-12-0610:10roklenarcicI guess I’ll see where this is going#2019-12-0610:22ikitommiI see. Need to think about this when at computer.#2019-12-0610:24roklenarcic@rschmukler yesterday I hit this snag https://clojurians.slack.com/archives/CLDK6MFMK/p1575567535304400#2019-12-0610:25roklenarcicyou made a patch#2019-12-0610:25roklenarcicI found another problem#2019-12-0610:26ikitommiIt was @rschmukler's patch#2019-12-0610:26roklenarcic(m/encode [:sequential {:encode/string (constantly {:leave #(clojure.string/join "," %)})}
string?]
["A" "B" "C"]
mt/string-transformer)
Error printing return value (NullPointerException) at clojure.core/map$fn (core.clj:2755).
null#2019-12-0610:27roklenarcicso if I wrap it in :leave I get NPE#2019-12-0610:27roklenarcicI know it was his patch that’s why I @ him#2019-12-0610:31ikitommiplease report all issues, clearly need to fix or rethink this. If you have suggestions how to make the transforming good with/without interceptors, while supporting the schema property based extension (serializable), I'm all ears.#2019-12-0610:47roklenarcicmy question before thinking about this is what is the desired logic for this part:
• if multiple transformers define encoder/decoder for same symbol/schema what is desired, last one wins?
• if schema use-site property map defined encode decode keys, what is the desired interaction with existing encode decode operation already defined by schema type itself, full override? does it also override the construction in -transformer?
#2019-12-0615:49rschmukler@roklenarcic Try the latest of that patch. Fixed that issue.#2019-12-0615:49rschmuklerThat was actually a lingering issue from a work-around that I did because fwrap wasn't skipping on non-collection input. Should be good now#2019-12-0620:05ikitommimy 2 cents:
• transformer should not have only one -transformer-name, but instead, a chain of names. strip-extra-keys should have an unique name
• with mt/transformer, one can create a chain of transformations, wehre both names and encoders & decoders are chained together and the chain of names is used for schema property based lookups. Any schema-defined will override the one defined in the transformer
• encoders and decoders are defined as a transformation function of type schema opts => IntoInterceptor. This allows any transformation function to access the schema in question at “compile time”, allowing fast “drop extran keys” etc.
• interceptor is to describe a transformation step. There is a IntoInterceptor Protocol, which is extended for a function (maps to :enter), a map, Interceptor Record or a chain of IntoInterceptor . Malli could use sieppari later, as it already defines all of these.#2019-12-0620:06ikitommiIn most cases, users don’t have to worry anything about the interceptor machineny, just use plain functions and it works. When you need more batteries on what happens when, you can either
a) start using interceptors (enter & leave)
b) add more steps into the transformation chain
c) both#2019-12-0620:07ikitommiSimplest thing:
(m/decode
[string? {:decode/string '(constantly #(str "olipa_" %))}]
"kerran" mt/string-transformer)
; => olipa_kerran#2019-12-0620:07ikitommiWith Interceptor:
(m/decode
[string? {:decode/string '(constantly {:enter #(str "olipa_" %)
:leave #(str % "_avaruus")})}]
"kerran" mt/string-transformer)
; => "olipa_kerran_avaruus"#2019-12-0620:08ikitommiA custom transformaton chain:
(m/decode
[:and
{:decode/before '(constantly inc)
:decode/after '(constantly (partial * 2))}
int?]
1
(mt/transformer {:name :before} {:name :after}))
; => 4#2019-12-0620:10ikitommiboth:
(m/decode
[:and
{:decode/before '(constantly {:enter inc
:leave (partial * 2)})
:decode/after '(constantly {:enter (partial * 4)
:leave inc})}
int?]
1
(mt/transformer {:name :before} {:name :after}))
; => 18#2019-12-0620:11ikitommiusing a chain:
(m/decode
[:and
{:decode/before '(constantly [inc inc #(* % 2) inc])}
int?]
1
(mt/transformer {:name :before} {:name :after}))
; => 7#2019-12-0620:30ikitommiAs we’ll soon lose the Slack History, wrote it down as issue: https://github.com/metosin/malli/issues/136#2019-12-0620:32ikitommialso, idea to remote the schema opts => interceptor intermediate step in favor of reitit-style interceptors that have a separate compile-step. Sounds even more complex, but actually would simplify things.#2019-12-0620:32ikitommiwith it, the simple case:
(m/decode
[:and
{:decode/after 'inc}
int?]
"1"
(mt/transformer {:name :before} mt/string-transformer {:name :after}))
; => 2#2019-12-0620:33ikitommithe complex case:
;; mounts only if there :multiplier property in schema
(def multiply-interceptor
{:name ::multiply
:compile (fn [schema _]
(if-let [multiplier (:multiplier (m/properties schema))]
(partial * multiplier)))})
;; the interceptor does not mount as the `:multiplier` is missing:
(m/decode
[:and
{:decode/after multiply-interceptor}
int?]
"2"
(mt/transformer {:name :before} mt/string-transformer {:name :after}))
; => 2
(m/decode
[:and
{:decode/after multiply-interceptor
:multiplier 10}
int?]
"2"
(mt/transformer {:name :before} mt/string-transformer {:name :after}))
; => 20#2019-12-0620:35ikitommi… and for the question:
> does it also override the construction in -transformer
no.#2019-12-0700:27roklenarcichm ok… but should the`-transformer` function return a function that returns transform function? Because so far it just seems that it brings a lot of constantly use#2019-12-0700:29roklenarcicI see so much constantly use that I wonder what is the other case (i.e. non-constantly)#2019-12-0703:58ikitommi-transformer already sees the schema, so it can return directly an .... interceptor? The later comment shows how we can get rid of vobstantly in the schema properties too: https://github.com/metosin/malli/issues/136#issuecomment-562731476#2019-12-0711:15ikitommithe current internal vocabulary is quite messy, would these be good names for the future:
* method :decode :encode
* stage :enter :leave
* context :json :string :before :after ...#2019-12-0712:05ikitommi:enter/:leave and chain of contexts enable same kind of things. Wondering do we need them both…
(def transformer (mt/transformer {:name :before} mt/string-transformer {:name :after}))
(m/decode
[:and
{:decode/after 'inc}
int?]
"1"
transformer)
; => 2
; :enter
; :before
; :strip-keys
; :string => ->int "1" => 1
; :after => inc 1 => 2
; :leave
; :after
; :string
; :strip-keys
; :before
(m/decode
[:and
{:decode/before {:leave 'inc}}
int?]
"1"
transformer)
; => 2
; :enter
; :before
; :strip-keys
; :string => ->int "1" => 1
; :after
; :leave
; :after
; :string
; :strip-keys
; :before => inc 1 => 2 #2019-12-0716:51ikitommiwelcome @viesti simple_smile#2019-12-0716:52viesti👋#2019-12-0716:53ikitommiDid a small spike, seems to work, so PR out. Simplifies the Transformer Protocol and enables chaining of tranformers (instead of overriding schema-based encoder & decoders like before). Also, better naming of things https://github.com/metosin/malli/pull/137#2019-12-0716:53ikitommi(m/decode
[int? {:decode/before '(constantly {:leave inc})
:decode/after '(constantly (partial * 2))}]
"10"
(mt/transformer
{:name :before}
mt/string-transformer
{:decoders {'int? (constantly inc)}} ;; anonymous
{:name :after}))
; => 23
;; :enter
;; :string "10" => 10
;; anonymous 10 => 11
;; :after 11 => 22
;; :leave
;; :before 22 => 23#2019-12-0917:11rschmukler@ikitommi just confirming that with the changes, a custom :string/decode on a schema entry will still fully overwrite the default mt/string-transformer for the type, yes? I think that escape hatch is quite useful.#2019-12-0917:12ikitommi@rschmukler yes, will override, no changes there#2019-12-0917:13rschmuklerWonderful! Reading through the code but it looks great so far#2019-12-0917:13ikitommijust chains all the different names.#2019-12-0917:14rschmuklerHmmmm - do we even need transformer-chain protocol method?#2019-12-0917:14rschmuklerIs it just for introspection?#2019-12-0917:14rschmuklerI guess we're using it to assemble the interceptor chain#2019-12-0917:14ikitommioh, good point.#2019-12-0917:15rschmuklerBut I think we could move all of that into the transformer function and then simplify Transformer to a single value-transformer method, which could be nice#2019-12-0917:18rschmukler@ikitommi do you have any thoughts on https://github.com/metosin/malli/pull/132#2019-12-0917:19rschmuklerIt ended up being critical for my application (the ability to encode the renaming of keys for future transformers) but I'm open to alternative approaches if you've got other ideas you want me to explore#2019-12-0917:19rschmuklerPart of me thinks using metadata is odd - part of me thinks that this is the perfect use case for it#2019-12-0917:20rschmuklerie. it is metadata about how the new value was created... so maybe it makes sense?#2019-12-0917:21rschmuklerOne thing we may have to do though, if we end up sticking with that API, is merge the metadata about transformers as we compose them#2019-12-0917:28rschmuklerAlso, random aside, but malli is Finish, correct?#2019-12-1006:30eskosIt’s Finnish for model (in both shape, example and fashion model sense), yes 🙂#2019-12-1006:32eskoshttps://www.sanakirja.org/search.php?q=malli&l=17&l2=3#2019-12-1007:13Toni Vanhalaand mälli is chewing tobacco 😉#2019-12-1010:08eskosor heavy impact 🙃#2019-12-1016:05roklenarcicis there some way to have a schema like `
s/keys*
#2019-12-1016:05roklenarcicNamely I’d like to generate sample function arguments, so I need a schema that describes function input#2019-12-1016:06roklenarcicbut the function is defined as [arg1 arg2 & {:keys [arg3 arg4]}#2019-12-1016:23ikitommi@roklenarcic there are no sequence schemas like that, yet. Feature Request welcome.#2019-12-1016:24ikitommidefinetely interested in having that, ideas on impl also welcome.#2019-12-1016:27ikitommi@rschmukler haven't had time to check the rename keys, will do that when next time doing anything for malli. Good thing is that you can depend your fork that's to deps.#2019-12-1017:54rschmukler@ikitommi checking your latest recommendation but I think I'll be able to break it 😛#2019-12-1017:56rschmuklerAh, I think I figured it out!#2019-12-1018:37rschmukler@ikitommi updated https://github.com/metosin/malli/pull/132 with something less abstract so you can see why it's difficult as currently implemented#2019-12-1018:42ikitommithanks, will try to find an elegant way to resolve.#2019-12-1106:03ikitommiChecked the code., the build-transformer doesn’t work correctly. I think the same problem is with all sequential schemas: the chain in :leave shoud be in reverse order, so first apply the (child) value-transformers, then keys and last the map. This way, when changing the keys in :leave , it’s the last thing that happens.#2019-12-1106:06ikitommiAlso, all the three type of transformers should be created outside of the build-transformer fn. Current impl creates all three two times (both in enter & leave) causing potential state-bugs with -value-transformerimpls as the enter & leave have different instances attached to them.#2019-12-1106:06ikitommiI’ll write an issue, PR welcome.#2019-12-1106:07ikitommiAfter that, this is the way to rename keys locally on encode:
(m/encode
[:map
[:foo [:map
[:foo int?]
[:bar int?]]]
[:bar int?]]
{:foo {:foo 2 :bar 1} :bar 3}
(transformer
{:encoders {'int? (constantly inc)
:map (constantly {:leave #(set/rename-keys % {:foo :oof :bar :rab})})}}))
; => {:oof {:oof 3, :rab 2}, :rab 4}#2019-12-1106:08ikitommiafter the upcoming interceptor :compile, will reduce to:
(m/encode
[:map
[:foo [:map
[:foo int?]
[:bar int?]]]
[:bar int?]]
{:foo {:foo 2 :bar 1} :bar 3}
(transformer
{:encoders {'int? inc
:map {:leave #(set/rename-keys % {:foo :oof :bar :rab})}}}))
; => {:oof {:oof 3, :rab 2}, :rab 4}#2019-12-1106:08ikitommiwhich will, I hope, solve your case @rschmukler elegantly enough?#2019-12-1115:25ikitommiwould be interesting to see how malli would work on the dev-tooling space. Related to https://www.reddit.com/r/Clojure/comments/e95pr5/data_structure_shape_hints_and_intellisense/#2019-12-1115:54rschmukler@ikitommi nice! Good catch on build-transformer being called twice and its impact on local state! Changing the order of the chains is also perfect - that'll make it much more like a postwalk on leave#2019-12-1118:04rschmukler@ikitommi I've got the first prototype of my defn-spec / ghostwheel clone. For now I'm doing it as a separate project (called aave). Will be releasing a first version shortly#2019-12-1118:06rschmuklerThe syntax is abstractable though, so you can use schema style too if you want. Things that it currently supports are purity detection, configurable hooks for input and output validation at both compile time and run time. Next up is benchmarking.#2019-12-1118:06rschmuklerI also thought it'd be cool to be able to do something like#2019-12-1118:08rschmukler(defn add-name-key
[a]
[:map => (+ a [:name string?])]
(assoc a :name "Bob")
Where + symbol basically expands out to "merge schema from parameter a "#2019-12-1118:08rschmuklerMight be overkill haha - but it could enable deep compile-time tracking of types#2019-12-1118:13ikitommi@rschmukler sounds awesome, and what a name! 👻#2019-12-1118:13rschmukler😄 I'm glad you like it#2019-12-1118:14bedersok, let's try this again. How would you write a function that returns a malli spec to test if a string is of length n?#2019-12-1118:15bedersI'm getting this far:#2019-12-1118:15beders(defn fixed-length? [n]
[:and 'string? [:fn `(~'fn [~'val] (~'= ~n (~'count ~'val)))]]
)
#2019-12-1118:15bedersthat gives me:
(preds/fixed-length? 10)
=> [:and string? [:fn (fn [val] (= 10 (count val)))]]
#2019-12-1118:18rschmukler@beders Do you need it to all be quoted? Malli should take care of that for you. if not, you can just do:
(defn fixed-length? [n]
[:and string? [:fn #(= n (count %)]])
#2019-12-1118:18bedersbut my code doesn't look very nice. Lots of quoting to prevent backtick from full name resolution#2019-12-1118:19bedersyour version captures n#2019-12-1118:19bedersI think#2019-12-1118:21beders(defn fixed-length? [n]
[:and string? [:fn #(= n (count %))]])
=> #'sql-to-malli.integration/fixed-length?
(edn/write-string (fixed-length? 10))
=>
"[:and string? [:fn #object[sql_to_malli.integration$fixed_length_QMARK_$fn__10416 0x576cee60 \"
#2019-12-1118:21rschmuklerAh I suppose that's true!#2019-12-1118:21bedersif I don't use the extension unquote-quoting, I get the fully resolved names#2019-12-1118:22rschmuklercan you do#2019-12-1118:22beders(defn fixed-length? [n]
[:and 'string? [:fn `(fn [val] (= ~n (~'count val)))]]
)
(fixed-length? 10)
=> [:and string? [:fn (clojure.core/fn [clojure.core/val] (clojure.core/= 10 (count clojure.core/val)))]]
#2019-12-1118:22bederswhich kinda works, just not very readable#2019-12-1118:23bederstrying to stick with the data-only mantra here#2019-12-1118:23bedersso no capturing#2019-12-1118:23rschmuklerBut you need your schema to be serializable?#2019-12-1118:24bedersI'm working on a library that takes a SQL database schema and spits out malli#2019-12-1118:24rschmuklerI'd consider adding it to your registry using fn-schema#2019-12-1118:24bedersI can probably do without serialization#2019-12-1118:24ikitommiyou can single-quote the fn#2019-12-1118:24bedersif I single-quote the fn, I'm capturing n#2019-12-1118:24rschmukler@ikitommi he needs to get the 10 in there though#2019-12-1118:25rschmuklerbut with a custom registry, you could just do [:and 'string? [:fixed-length 10]]#2019-12-1118:25bedersyes, adding it to the registry might be the only sane choice for now#2019-12-1118:26bedersthat would make my library a runtime dependency#2019-12-1118:27bedersI was hoping I could run this on a database schema, copy the resulting malli schemas and then run with them#2019-12-1118:27bederslooks like I need to decide on various trade-offs#2019-12-1118:27ikitommia :fn variant that takes extra args from properties?#2019-12-1118:27bedersjust wanted to get some ideas on code generation#2019-12-1118:28bedersthat make for readable schemas#2019-12-1118:28bedersI like [:fixed-length 10] though. Very readable#2019-12-1118:30bedersthanks for the ideas!#2019-12-1118:30rschmuklerOh you know#2019-12-1118:31rschmukleryou might be able to do#2019-12-1118:32rschmuklerNope, never mind 😄#2019-12-1118:50ikitommiJSON Schema defines extra attributes for numbers, strings and arrays. Malli could have those too?
[string? {:min 10, :max 10}]#2019-12-1118:51ikitommiAlso, malli schema properties should be defined with malli schemas to get nice docs & validation of those too.#2019-12-1118:54ikitommi@beders if the runtime dependwncy would be ok, you could also just use the Schema Var instead of registry:
[my-lib.schemas/fixed-lenght 10]#2019-12-1118:58bedersGood point. I mean, technically, this works too:
(m/validate [:re #"^.{4}$"] "bubu")#2019-12-1118:58bedersbut probably a bit overkill 😉#2019-12-1118:58ikitommithe fn can also be a string btw, so this should work (injection warning!!):
[:and string? [:fn (str "#(= " n " (count %))")]]#2019-12-1119:00bedersah, didn't know that. Thx!#2019-12-1119:02ikitommilooking forward to seeing the lib.#2019-12-1201:42bedersit looks like m/fn-schema doesn't allow for any other parameters (or children)#2019-12-1201:43bedershere's what I tried:
(defn max-length?
([n s]
(and (string? s) (< (count s) n)))
)
(def default-registry (merge m/default-registry {:max-length (m/fn-schema :max-length preds/max-length?)}))#2019-12-1201:44bedersI then should be able to say this, right?
(m/validate [:max-length 4] "bubu" {:registry default-registry})#2019-12-1201:45bedersbut alas
#error{:cause ":malli.core/child-error",
:data {:type :malli.core/child-error, :data {:name :max-length, :properties nil, :children (4), :min 0, :max 0}},#2019-12-1201:45bederswhich - when checking the code - happens in fn-schema#2019-12-1201:46bedersit seems -partial-fn-schema allows for additional parameters, but is private#2019-12-1201:46bedersalso, fn-schema doesn't pass on properties#2019-12-1201:48bedersany hints on how to register custom predicates with params would be awesome#2019-12-1205:41JohanIs Malli a good place to handle default value ?#2019-12-1207:09ikitommi@johan178 sure. could be just :default property and a pre-defined transformer to use that for missing values. metosin/schema-tools has just about that.#2019-12-1207:10ikitommineed to add capability to transform all schemas, so need a small development before it can work. Please raise an issue.#2019-12-1219:47rschmuklerHey @ikitommi I'm in the process of fixing up transforms for independent leaves and enters (and reverse order). I have a few questions for you while I'm in here...#2019-12-1219:51rschmukler1. For the collection transformer, right now fwrap auto-coerces things into the appropriate type. This happens after the value-transformer is called currently - meaning that I don't think it's possible for the value-transformer to change it's collection type on encode because it'll always be coerced back. Should we keep this behavior?
2. the map-of transformer currently calls transform keys and transform values at the same time as it iterates through the map. I think it's fine to keep this for enter and leave? ie. it'll basically be enter: transform-map -> transform k 1 -> transform v 1 -> transform k 2 -> transform v 2 and leave: transform k 1 -> transform v 1 -> transform k 2 -> transform v 2 -> transform map
3. The collection transformer currently does a single step of building the collection and mapping the values. We probably want to do enter: coll-transform -> transform children and leave: transform-children -> transform#2019-12-1221:03ikitommi1) I think the fwrap should be removed and moved under Transformer decoders. e.g. as the arrays are always mapped to Clojure vectors in JSON decoding (by json parsers in both clj & cljs), with mt/json-transformer there should be no transformation needed for :vector, only for other sequence types. The currently silly mt/collection-transformercould have decoders for all collection types.#2019-12-1221:03ikitommi2 & 3 👍#2019-12-1221:04rschmuklerRe. 1 I want to land this PR w/o that then if you don't mind. It's already huge#2019-12-1221:04ikitommisure#2019-12-1221:05rschmuklerThe other thing that we don't do much, that I'm going to add, if you think it's a good idea, is ensuring that the type doesn't change between transforms#2019-12-1221:05rschmuklerie. if a map encodes itself to an integer, we still try and call reduce-kv on it for the key and value transformers#2019-12-1221:05rschmuklerWe likely shouldn't do that#2019-12-1221:06rschmuklerActually - I'm going to keep this focused at just the changes to fix #140 and the encode decode order - we can add more later#2019-12-1221:07ikitommibtw, the 3 might have a big effect on the perf?#2019-12-1221:08rschmuklerI'm still using transducers where possible#2019-12-1221:08ikitommidepending on how the collection is transformed, retaining the type automatically or not#2019-12-1221:08rschmuklerand using fempty etc#2019-12-1221:08ikitommicool. Sorry, Need to go.#2019-12-1221:08rschmuklerI tried to be mindful of the performance implications - but a close look is definitely a good idea!#2019-12-1221:09rschmuklerCool, PR incoming#2019-12-1221:09rschmuklerCatch you later!#2019-12-1304:53JohanCan a defined transformer access the schema ?
So given [:map [:key {:option :cool} keyword?]]
Can a transformer access the [:key {:option :cool} keyword?]] data ?#2019-12-1306:12ikitommi@johan178 yes, the current transformer is of type schema opts => value => value, so:
[:map {:decode/string (fn [schema opts]
(let [cool (:option (m/properties schema))]
(fn [value] …)))}]#2019-12-1306:13ikitommithe syntax will change before hitting first public to:
[:map {:decode/string {:compile (fn [schema opts]
(let [cool (:option (m/properties schema))]
(fn [value] …)))}}]#2019-12-1306:14ikitommihttps://github.com/metosin/malli/issues/136#issuecomment-562731476#2019-12-1316:35rschmukler@ikitommi Thanks for the quick PR review! Would you be open to me adding .nrepl-port and .dir-locals.el to gitignore? Alternatively if you want me to commit a .`dir-locals.el` it would make it so that new people contributing would automatically adhere to your style preferences#2019-12-1316:46ikitommi@rschmukler sure, go ahead, either way#2019-12-1316:46rschmukler:thumbsup:#2019-12-1317:29eskos.dir-locals.el is only relevant to cider users, so I wouldn't commit them. In general tooling/IDE config files shouldn't live in source repos 😛#2019-12-1317:48rschmukler.dir-locals.el is relevant to all emacs users#2019-12-1317:49rschmuklerBut I'm happy to not commit it - just could save PR time of people having to adjust their formatting to the project style#2019-12-1317:59rschmuklerhttps://github.com/teknql/aave - First rendition of Aave!#2019-12-1318:08ikitommiagree on Esko for now, so let’s ignore the files.#2019-12-1318:09ikitommiCongratz on Aave!! I’ll add it to README as soon as the PRs are merged (just did the interceptor :compile, will merge it after yours @rschmukler).#2019-12-1318:09rschmuklerWoo! Thanks! Love the momentum we've got going on!#2019-12-1318:22rschmukler@ikitommi What're your thoughts for potentially adding support for a dynamic variable for either full on malli options, or perhaps something like *registry-extensions* which is usually nil but is automatically merged into default-registry when set. I sometimes find myself having to thread malli options through things, or wrapping malli functions with my own namespace with a custom registry, etc. Since it's mostly for compile-time code anyway, the performance implications shouldn't matter much. Part of me feels like it might be too "magical" but the other part makes me feel like it could improve ergonomics#2019-12-1318:33ikitommineed to think about that.#2019-12-1318:34ikitommi@rschmukler do you have plan to fix the 142? (the is the x bug IMO + the formatting)#2019-12-1318:35rschmuklerI think I already did no?#2019-12-1318:35rschmuklerI force pushed updates an hour or two ago#2019-12-1318:36rschmuklerLet me know if it needs anything. The x bug wasn’t caught by tests because map-of tests were disabled so I also fixed those #2019-12-1318:37ikitommioh, the review panel just doesn’t show the updated. my bad. will merge.#2019-12-1318:38rschmuklerAwesome!#2019-12-1318:40ikitommicould you check https://github.com/metosin/malli/pull/146 ?#2019-12-1318:44rschmuklerYep, will take a look now#2019-12-1318:53rschmuklerLooks good! Sorry I missed some indents#2019-12-1319:09ikitomminp, off to weekend. cheers.#2019-12-1510:57ikitommiAbout to add :default-decoder and :default-encoder to Transformer options: used if nothing else matches. Makes things like “adding defaults” easy. Comments welcome on that: https://github.com/metosin/malli/issues/143#issuecomment-565798326#2019-12-1520:30ikitommiAnd here’s the PR. Rewrote the malli.core naming to be more corerent, more reuse in chaining & collection schemas now accept nil in transformations. https://github.com/metosin/malli/pull/148#2019-12-1617:25rschmukler@ikitommi awesome cleanup on malli.core. Code looks 😍#2019-12-1617:31rschmuklerJust rebased https://github.com/metosin/malli/pull/147#2019-12-1617:42ikitommiactually, the logic to convert from keywords should be somewhere. JSON Parsers keywordize the keys. If keys are defined to be int?, They are seen as :12 etc. The string->int doesn't work here#2019-12-1617:43rschmuklerAh! I can look into re-adding that to the json transformer then#2019-12-1617:43ikitommi.. and string too#2019-12-1617:44rschmuklerOkay, should I just add it to the string transformer then?#2019-12-1617:44rschmukler@ikitommi I think we should consider dropping the schema-based type checks and instead leave that to the transformer functions. ie. sometimes we check (if (or (map? x) (nil? x)) (transform x)) - especially on decode, it might not be the same shape at all...#2019-12-1617:44rschmuklerConsider a JSON body that returns [x y z] and wanting to decode that into a map... or similarly wanting to use malli to decode CSV rows into maps#2019-12-1617:45rschmukler(decode [:map {:decode/row (fn [row]
{:name (nth row 0)
:age (nth row 1)})}
[:name string?]
[:age int?]]
["Bob" "42"]
(malli.transform/transformer
malli.transform/string-transformer
{:name :row}))#2019-12-1617:45rschmuklerIdeally that'd work#2019-12-1617:47rschmuklerI think it should be the transformer's responsibility to handle (or not handle) arbitrary data#2019-12-1618:04rschmukler@ikitommi re-added the coercion behavior to string + json transformer. Still marked the commit as breaking since it'll break people who were relying on the keyword->string coercion in their transformers... but I updated the message + commit body to tell as much.#2019-12-1618:44ikitommi@rschmukler agree on pushing the checks to transformers. Currently, the check is only done before the whole chain. If the first step changes the type, with the current solution, the next will blow anyway.#2019-12-1618:45ikitommi147 looks good. I was told not to pull in before the EPL2 license is in place, need still few approvals.#2019-12-1618:47ikitommiI think the current :malli.core/map-key target (and impl) can be removed - same can be done with anonymous transformer with :map target.#2019-12-1619:25rschmukler@ikitommi Alright sounds good. I think I've got a elegant solution to the transformer chain checks#2019-12-1621:02ikitommihttps://github.com/metosin/malli/pull/149#2019-12-1622:22rschmukler@ikitommi https://github.com/metosin/malli/pull/150 😄#2019-12-1716:26rschmuklerAny word on when the code freeze will end?#2019-12-1718:41miikkaAs soon as hear back from everyone. I've got 4/6 confirmations so far.#2019-12-1721:05rschmuklerWoo!#2019-12-1816:15rschmukler@ikitommi I haven't looked at the code at all yet, but how difficult do you think it would be to extend the malli.provider to be pluggable / use transformers? ie. Ideally I could specify a transformer and it'd try transformations in there, and it'd check types that exist in the registry when called#2019-12-1816:19rschmuklerLooking at the code - doesn't look like it'd be too hard to make it take a transformer!#2019-12-1816:20rschmuklerThe only thing would be handling the fact that multiple transformers could pass and then having to keep track of which one was most successful#2019-12-1816:20rschmuklerBut we are kinda already doing that with the type anyway#2019-12-1909:05miikkaMalli is now under EPL-2.0 and the code freeze has ended.#2019-12-1913:43pezI should have looked for a malli channel, of course!#2019-12-1915:32ikitommi#2019-12-1915:32ikitommi#2019-12-1915:33ikitommiwelcome @pez! looking forward to your validation lib 🙂#2019-12-1915:51pezAwesome. I am using 1 right now. It was 3 I was trying to do but didn't figure out about the :error/path property. Will try that now.#2019-12-1915:53pezDon't hold your breath about that lib. 😃 But who knows, we do have many kinds of form scenarios in our app so maybe I get reason to really figure this out.#2019-12-1917:42rschmuklerhttps://github.com/weavejester/integrant/pull/60 - fingers crossed 🙂#2019-12-2006:55ikitommi@rschmukler malli.provider, transformer, so it could if there is a use case for it. Some kind of extension mechanism would be good anyway, adding inferring to :tuple and :multi currently require a lot of internal refactoring. Not easy to add inferrers to own schemas.#2019-12-2006:57ikitommias it's not complete yet, would be good to add the missing inferrers and options to controls those and see what kind of architecture evolves out of those.#2019-12-2007:02ikitommimaybe options like {:infer {:multi {:stats ...,. :schema ..., :opts {:dispatch #{:type 'first}, :min 2, :propability 0.90}}},#2019-12-2007:04ikitommie.g. callback to participate in extracting stats out of samples, and converting those into schemas + opts for what kind of multis to look and how many samples needed#2019-12-2007:06ikitommimight be better to start with just options and hard-code the impl, and try to extract the callback mechanism later.#2019-12-2016:27rschmukler@ikitommi the other thing I was thinking about is the ability to code options. ie. imagine a LocalDateTime parser that has an property of :date/format "MM/dd/yyyy" - ideally it'd be good to specify "try these formats when you try to parse things as a date" etc#2019-12-2016:27rschmuklerStill thinking about how it could look#2019-12-2018:29ikitommiidea: If the inferrer is good, there could be a tool that records all args and return values for functions and infers defn-schemas out of those. I think @ambrosebs did something similar with spec (using core.typed)#2019-12-2018:38pezLovely idea!#2019-12-2023:42borkdude@ikitommi that's awesome. this information can then also be used to generate clj-kondo type annotations#2019-12-2114:12ikitommi2 months to ClojureD, would be awesome to have some clj-kondo integration for that.#2019-12-2114:13ikitommiAlso, to finalize this…#2019-12-2114:17ikitommidon’t know if it’s problem in fipp or in my cursive setup, but some some broken ansi escape codes threre.#2019-12-2116:05borkdude@ikitommi This is the format the clj-kondo currently understands: https://github.com/borkdude/clj-kondo/blob/master/doc/types.md
Any tool that can generate that format from whatever data available can integrate with clj-kondo#2019-12-2119:07ikitommiwell, that’s looks simple.#2019-12-2716:12ikitommidid a quick cleanup, without too much thinking, so comments welcome, https://github.com/metosin/malli/pull/152#2019-12-2809:38ikitommireitit coercion with malli 90% done#2019-12-2818:28rschmukler🔥🔥🔥#2020-12-3003:41JohanI'm trying to implement the ideas developed on this talk with Malli.
Anyone else is splitting attributes schemas and selections ?
source: https://www.youtube.com/watch?v=YR5WdGrpoug#2020-12-3008:54ikitommi@johan178 some discussion here: https://github.com/metosin/malli/issues/95#2020-12-3009:04ikitommiAlso: https://github.com/metosin/malli/issues/14#2020-12-3009:08ikitommithat said, it would be trivial to write a custom select schema, allowing serializable selects:
[:select
[:map [:x int?] [:y int?]]
[:x]]
#2020-12-3110:57borkdudeDoes malli have something like s/conform for s/cat?#2020-12-3111:11ikitomminot yet, but it will come. Can't recall if there was an issue of conform yet, or just discussed#2020-12-3112:44ikitommi12 added lines later:
(explain
[:map {:closed true}
[:xy
[:map {:closed true}
[:x int?]
[:y int?]]]]
{:xy {:x 1, :y 2, :EVIL "LYN"}
:DARK "ORKO"})
;{:schema [:map {:closed true} [:xy [:map {:closed true} [:x int?] [:y int?]]]],
; :value {:xy {:x 1, :y 2, :EVIL "LYN"}, :DARK "ORKO"},
; :errors (#Error{:path [2 1],
; :in [:xy :EVIL],
; :schema [:map {:closed true} [:x int?] [:y int?]],
; :type :malli.core/extra-key}
; #Error{:path [],
; :in [:DARK],
; :schema [:map {:closed true} [:xy [:map {:closed true} [:x int?] [:y int?]]]],
; :type :malli.core/extra-key})}#2020-12-3112:45ikitommithat should contain all the needed info to get spell-spec grade error messages.#2020-12-3112:46ikitommie.g. :in points to the actual key and :type can be used to format suggestions on misspelled keys etc.#2020-12-3113:11ikitommihttps://github.com/metosin/malli/pull/156#2020-12-3113:28ikitommialso:
(-> [:map {:closed true} [:x int?]]
(m/explain {:x 1, :extra "key"})
(me/humanize))
; => {:extra ["disallowed key"]}#2020-01-0208:52ikitommiProgramming with Schemas: https://github.com/metosin/malli/pull/158#2020-01-0208:55ikitommiQuite happy how simple Schema transformations are with the current design, recursively closing all Schemas:
(defn closed-schema
"Closes all :map Schemas recursively."
[schema]
(m/accept
schema
(m/schema-visitor
(fn [schema]
(if (= :map (m/name schema))
(update-properties schema assoc :closed true)
schema)))))#2020-01-0313:12ikitommicomments welcome on handling extra keys with maps: https://github.com/metosin/malli/issues/43#2020-01-0515:28roklenarcicthere;’s a lot of branches in repo that have been merged and not deleted#2020-01-0521:07ikitommisure there is, I guess there is no automation removing those after merge/rebase.#2020-01-0611:20eskoshttps://help.github.com/en/github/administering-a-repository/managing-the-automatic-deletion-of-branches#2020-01-0611:20eskosYou do need to delete the old merged branches manually though; Google will find you some bash scripts for that.#2020-01-0521:09ikitommiwould like to break thing a bit: mt/strip-extra-keys-transformer should be a function of options => Transformer , not a Transformer. Question: should it remove extra keys by default:
a) from all maps (open or closed)
b) only from closed maps#2020-01-0521:09ikitommiwill add a option to control this, but leaning on a) as the default.#2020-01-0522:56roklenarcichm… how do I make a transformer that uses a schema property as param to transform function#2020-01-0522:57roklenarcicyou have this showcased in the :compile option example#2020-01-0522:58roklenarcicbut I’d like to look up something like that `
:math/multiplier
in the default transformer function#2020-01-0522:59roklenarcice.g. I have datetime schema and it’s transformer would like to check if schema had :format property#2020-01-0523:00roklenarcicthe :compile thing requires one to add it to every use-site#2020-01-0523:02roklenarcicoh, I see how it’s done elsewhere, ignore that#2020-01-0611:10ikitommiyeah, the default-value-transformer uses schema properties and mounts for all schemas.#2020-01-0618:26ikitommiBreaking transformer api, hopefully making it better: https://github.com/metosin/malli/pull/162#2020-01-0618:26ikitommicomments welcome on that#2020-01-0618:29ikitommiThe goal is to have a unified api for creating Transformers: all are created via a function, instead of mixing functions and actual Transformer instances. So, instead of:
(mt/transformer
mt/string-transformer
mt/strip-extra-keys-transformer
(mt/key-transformer
{:decode #(-> % name (str "_key") keyword)
:encode #(-> % name (str "_key"))}))
we should do:
(mt/transformer
(mt/string-transformer)
(mt/strip-extra-keys-transformer)
(mt/key-transformer
{:decode #(-> % name (str "_key") keyword)
:encode #(-> % name (str "_key"))}))
… both work thou, thanks to auto-coercion of functions => Transformers.#2020-01-0618:32ikitommiin other libs (schema, spec-tools) the json-transformer|matcher are just values, so people most likely try that instead of call the function. Not 100% happy with having the auto-coercion, but here thought the developer experience is more important than having an one explicit (non-familiar?) syntax.#2020-01-0812:05ikitommimerged into master#2020-01-0912:08ikitommihei, someone suggested that malli (and reitit) should be sponsored by Clojurists Together, any insight what should be developed? Thanks anyway 🙇 https://www.clojuriststogether.org/news/q1-2020-survey-results/#2020-01-0917:21roklenarcicare you sure -composite-schema function -transformer works correctly? other schemas with children use -chain helper to make sure that :leave transformers are combined in reverse order than :enter , but this one does not#2020-01-0917:35ikitommibug? PR welcome.#2020-01-1109:10ikitommiUpcoming:
(-> (m/explain
[:map {:closed true}
[:orders boolean?]
[:deliver boolean?]]
{:orders true
:deliverz true})
(me/with-suggestions)
(me/humanize))
;{:deliver ["missing required key"]
; :deliverz ["likely misspelling of :deliver"]}#2020-01-1120:46roklenarcicAdded a couple of schemas I’m using to a public repo https://github.com/RokLenarcic/malli-schemas#2020-01-1211:08ikitommiNice, date-support is on malli backlog, need to figure out a way to make it work with cljs too.#2020-01-1215:13roklenarcicGenerally, transform operations do not validate that input passes the validator first, which is good for performance, but when you have an operation like :or how do you know which branch of OR to use for transformation? Currently you try every branch as transformation and return the first result that changes the value.#2020-01-1215:13roklenarcichowever this leads to this:#2020-01-1215:14roklenarcic(m/encode [:or
[string? {:encode/string #(.toUpperCase %)}]
[int? {:encode/string str}]]
1
mt/string-transformer)#2020-01-1215:14roklenarcicExecution error (IllegalArgumentException) at com.github.roklenarcic.malli-inline/eval17000$fn (form-init15839185010501867114.clj:2).
#2020-01-1215:14roklenarcicNo matching field found: toUpperCase for class java.lang.Long
#2020-01-1215:16roklenarcicnone of the built-in transformers will validate input and return the value itself if not the right type, which would make :or work correctly in this regard#2020-01-1215:28roklenarcicThe only realistic way to fix this would be to have :or do validate for each branch until it finds one that validates and then run that transform. This would perform as expected, I can prepare a PR.#2020-01-1218:48ikitommi@roklenarcic sounds right to me to select branch based on validate#2020-01-1218:49roklenarcicanother possible bug#2020-01-1218:49roklenarcicin about 5 different explainer functions in malli.core you have distance (if (seq properties) 2 1)#2020-01-1218:50roklenarcicif properties is missing then distance should be 1#2020-01-1218:50roklenarcicthe problem is that if someone puts {} as properties, then (seq properties) is false#2020-01-1218:51roklenarcicand it will say distance 1, when the form has [symbol {} children]#2020-01-1218:51ikitommitrue#2020-01-1218:51roklenarcicso the path is possibly wrong… I don’t know what the semantics of path is exactly#2020-01-1218:52ikitommiget-in to m/form of the schema with the error :in should point to schema#2020-01-1218:52ikitommineeded in pretty error reporting , which is wip.#2020-01-1218:52roklenarcicbut you see what I’m getting at… (seq properties) is nil for nil and {}, not good for counting elements in form#2020-01-1218:53ikitommiyes, that's currently wrong but should be easy to fix#2020-01-1218:53ikitommiGood thing that it's pre-alpha ;)#2020-01-1218:54ikitommiWill need malli at a project starting tomorrow, so good reason to push out first alpha too#2020-01-1220:19ikitommishould the schemas allow nil children? e.g. [:and int? nil pos-int?]. Currently :map allows, composite schemas don’t.#2020-01-1220:20ikitommie.g. [:map [:x int?] nil [:y pos-int?]] works, [:and int? nil pos-int?] doesn’t.#2020-01-1220:20ikitommiall core schemas should behave identically.#2020-01-1220:22ikitommialso, currently:
(m/form [:and {} int?])
; => [:and int?]
(m/properties [:and {} int?])
; => {}#2020-01-1220:23ikitommishould empty properties be removed (set to nil) already when schema is created from AST?#2020-01-1220:23ikitommiafter that:
(m/form [:and {} int?])
; => [:and int?]
(m/properties [:and {} int?])
; => nil#2020-01-1220:28ikitommimy suggestion:
• allow nil child schemas in all core schemas: easy to create programmatically (e.g. [:map [:x int?] (if require-y [:y int?])])
• treat empty properties as nil, allows (`[:map (if close-maps {:closed true}) [:x int?]]`)#2020-01-1220:38roklenarcicsure, sounds sensible#2020-01-1222:00roklenarciccan explainer fn return null? I thought it must return acc#2020-01-1223:09pithylessI agree all core schemas should behave identically; I would suggest not ignoring nil children, which gives more obvious semantics in cases like this: [:and int? nil pos-int?] (this would always be false, instead of implementing a special-case for nil that behaves differently than regular "falsey" semantics in Clojure). In the map example, I find it more intuitive to use malli.util/merge semantics to describe programmmatic optionality: (malli.util/merge [:map [:x int?]] (if require-y [:map [:y int?]]))#2020-01-1305:45ikitommifixed so that nil is allowed as properties and empty properties are treated as nil (and stripped from m/form).#2020-01-1305:51ikitommihttps://github.com/metosin/malli/pull/164#2020-01-1305:46ikitommiI don’t think nil should be treated as “always fail” in composite schemas. Currently, it will blow as nil is not a valid schema.#2020-01-1305:47ikitommialso, can’t strip all nils from children, as things like :enum might have actual valid value of nil.#2020-01-1305:49ikitommiSo, options being:
1. allow (and strip) nil in map & composite schemas, for convenience sake
2. blow up in case of nil as child schema
3. map nil to “always fail”#2020-01-1305:50ikitommithe malli.util merge is not that fun for deeply nested maps:
(malli.util/merge
[:map [:x [:map [:y [:map [:z int?]]
(if require-z2 [:map [:x [:map [:y [:map [:z2 int?]]))
#2020-01-1308:54roklenarciceach into-schema should probably decide for itself what to do with nil values#2020-01-1414:53ikitommimerged the spell-checking, https://twitter.com/ikitommi/status/1217096807316119553#2020-01-1520:06rschmuklerHey @ikitommi just a heads up that https://github.com/teknql/pablo exists for when / if you go to break malli into sperate libraries. It's still early days, but it's made to ease the burden of managing mono-repo-like libraries. Documentation is sparse at the moment, but I don't anticipate that you'll be slicing malli up too soon either#2020-01-1520:36ikitommiOh, that looks great! How do you setup the artifact version when pushing to clojars?#2020-01-1521:14rschmuklerTwo options (again, sorry on the sparse docs)#2020-01-1521:14rschmuklereither, you invoke it with an explicit :version via the repl / CLI (cli coming soon)#2020-01-1521:14rschmukleror, it automatically looks at your git status and uses the tag#2020-01-1521:15rschmuklerie. if git is tagged at 1.3.0 and not dirty, it's 1.3.0, if it's dirty then it's 1.3.0-SNAPSHOT#2020-01-1521:15miikkaHmh #2020-01-1521:15rschmuklerMight also make it so that if you're ahead of a tag (but not dirty) it does 1.3.0-SNAPSHOT too#2020-01-1521:15miikkaWhy not just use Leiningen?#2020-01-1521:16rschmuklerI ask myself that at least once a week 😉#2020-01-1521:17rschmuklerI'm hoping to get pablo to ultimately compile as a native image#2020-01-1521:18rschmuklerAlso, I think the biggest thing that tools-deps gets right is that it forces your project into a edn map - lein allows real programming to happen inside the project.clj file which can allow it to become quite the mess#2020-01-1521:19rschmuklerBut, I don't disagree that this is shoe-horning some lein-like functionality into tools deps#2020-01-1521:21miikkaYeah… my ideal setup right now would be actually to have Leiningen for the project management (because it already works) and tools.deps for dependency management. Too bad that lein-tools-deps doesn’t quite make it work. #2020-01-1521:23rschmuklerThe thing that kills me with tools-deps is that they didn't make a project.clj parser for when cloning via git... So if you attempt to depend on a git repo and they use lein and don't publish the xml file in the repo, you're SOL#2020-01-1521:24miikkaYeah#2020-01-1521:24rschmuklerIt feels very unpragmatic#2020-01-1521:24miikkaWell, you can’t correctly parse project.clj without running it. Although of course you could make it work in most cases. #2020-01-1521:25rschmuklerYeah, 1) you could naively parse it and cover 99%, and 2) sci now exists - so you could eval it at make it work in 99.9999% of cases#2020-01-1521:26miikkaI think tools.deps is designed in such a way that you can add new dep types. But it’s not nice if it’s not there out of the box#2020-01-1521:27rschmuklerYeah I think you're right - I think the coordinates can all be expanded on via multi-method. But then I'm not quite sure how you get it to load those multimethods before trying to resolve the local deps.edn file#2020-01-1521:28miikkaYeah#2020-01-1716:05theeternalpulseHi folks, starting to incorporate malli into my project, was wondering is there a mechanism to add metadata to a function so that input and output are automatically validated, or is all the validation meant to be done manually within the function?#2020-01-1716:15ikitommi@theeternalpulse two choises:
1. wait for https://github.com/metosin/malli/issues/125
2. try aave, https://github.com/teknql/aave#2020-01-1716:16theeternalpulseoh sweet#2020-01-1716:16ikitommigoal of malli is to be a complete schema-system, both function schemas and linting (integration to clj-condo) will be there, soon, I hope.#2020-01-1716:17theeternalpulse👍 thanks, I'll take a look at aave in the meantime.#2020-01-1716:18ikitommiany discussion of aave most welcome here, as @rschmukler is here too.#2020-01-1716:21rschmuklerHi @theeternalpulse! Aave is in its early days but it does exist! It's syntax is extendable so if you don't like the ghostwheel-like syntax, it wouldn't be too hard to add. The other thing that I want to add is the option for compile time testing of pure functions (so we refuse to compile, rather than wait for the user to call the function). If you have any feature requirements please feel free to open some issues#2020-01-2216:36ikitommithe m/accept will need extra parameters, maybe both :in and :path. Need to be able to better support schema mappings and inferring ui’s from mall schemas.#2020-01-2216:37ikitommiwill break all 3rd party Schema extensons, sorry.#2020-02-0417:36ikitommimaybe there should be default-options-provider, that can be overridden once?#2020-02-0417:36ikitommi, would allow easy plugging in of custom registeries, and other options how things work#2020-02-0417:37ikitommi(integrating malli into real projects, need custom global schemas without a hassle)#2020-02-0510:31borkdudeMaybe metosin wants to be mentioned here because they are using sci in malli? https://github.com/borkdude/babashka/issues/254#2020-02-0510:49ikitommidid. I guess you have logo already from clj-condo question 😉#2020-02-0510:54borkdudeyep, thanks#2020-02-0618:58eoliphantwill malli eventually do, for lack of a better term, a keyset coerce/conform? I try to use open keysets as much as possible, but always struggle the best way to handle stuff like the following
(-> {:a .. b:.. :c .. :d} ; say input from api call
validate ; we only care about :a,:b and :c but ignore :d
do-something-with-a-and-b ; fine with a s/keys [:a :b] or m/validate equivalent precondition
do-something-with-c ; fine with a s/keys [:c]
(select-keys [:a ::c]) ; or (transform my-schema)
(write-a-b-c-to-datomic) ; at this point I want a closed schema/spec precond, that could potentially narrow the keys on the way in? #2020-02-0619:05ikitommi@eoliphant would something like (mu/select-keys map-of-abc [:a :b]) be good? you can always make maps closed, to either fail on extra keys or to strip extra keys in transform#2020-02-0619:05ikitommie.g. select-keys, but for schemas#2020-02-0619:08eoliphanthmm yeah, perhaps strip extra on transform is the way to go? again, just been thinking about this, in terms of a structural coercion, but that might not be the best approach#2020-02-0619:13ikitommiI think it's the way to go: open schemas are great , but external systems like db's aren't always happy with extra stuff in. We are at least stripping data away at the boundaries.#2020-02-0619:13ikitommithere is malli.transform/strip-extra-keys-transformerfor this#2020-02-0619:16eoliphantespecially for stuff like datomic lets you do maps, etc all the way down, but it gets understandably upset if you pass in attributes it doesn’t know about 😉#2020-02-0720:42eoliphanthi quick question on error messages. I’m trying to do a custom message for each condition, so for your initial example, somehting like
(-> [:and
[int? {:error/message "Not int"}]
[:> 6 {:error/message "Less than 6"}]]
(m/explain 6))
That doesn’t work, nor does say this [[:> 6] {:error/message "Less than 6"}]] though
(-> [:and
[int? {:error/message "Not int"}]
[:> 6]]
(m/explain "a"))
is just fine#2020-02-0806:38ikitommiproperties are always (if defined) the first argument in the schema vector, so [:> {...} 6] should work#2020-02-0806:39ikitommi"hiccup syntax"#2020-02-0814:51eskosHow battle tested is the serializable function schemas feature of malli? I'm tinkering on a small side project of mine and realized malli would actually fit into it really well for the purposes of schema+coercion of data structures saving me the effort of doing those myself... 🙂#2020-02-0815:16ikitommi@suomi.esko heavy lifting done by https://github.com/borkdude/sci#2020-02-0815:17ikitommimalli should be in alpha around ClojureD#2020-02-1016:49ikitommistarted to add programmatic helpers to malli: https://github.com/metosin/malli/pull/172#2020-02-1016:50ikitommilooks like:
(malli.util/get-in
[:map {:title "test"}
[:x [:vector
[:list
[:set
[:sequential
[:tuple int? [:map [:y [:maybe boolean?]]]]]]]]]]
[:x 0 0 0 0 1 :y 0])
; => boolean?#2020-02-1016:51ikitommithere is a new Protocol LookupSchema , which is implemented for :map, :multi, :vector, :list, :set, :sequential, :tuple.#2020-02-1016:52ikitommi(malli.util/select-keys
[:map {:title "map"}
[:a int?]
[:b {:optional true} int?]
[:c string?]]
[:a ::extra])
;[:map {:title "map"}
; [:a int?]
; [:b {:optional true} int?]]#2020-02-1016:52borkdudeso kinda like the new spec2 thing, where you can select from your schema in different contexts?#2020-02-1016:54ikitommiwell, almost. like in schema-tools, going to use the clojure.core functions names. There could be malli.util/select to do things like spec2 does.#2020-02-1016:54ikitommi(-> [:map
[:a int?]
[:b string?]
[:c [:map
[:x int?]
[:y keyword?]]]]
(mu/select-keys [:a :c])
(mu/update-in [:c] mu/dissoc :y))
;[:map
; [:a int?]
; [:c [:map
; [:x int?]]]]#2020-02-1016:55borkdudenice#2020-02-1016:55ikitommithat’s the thing we have been using with prismatic schema for 3+ years now, happy with that 🙂#2020-02-1016:57ikitommiI’m not sure how the spec2 select works with nested sequential specs#2020-02-1016:58ikitommiin malli, I want to keep things explicit: you need to walk those over yourself.#2020-02-1016:58borkdudeme neither, I kinda lost track on spec2#2020-02-1016:58ikitommiLookupSchema
(-get [_ key default] (if (= 0 key) schema default)))
#2020-02-1016:59ikitommi^:--- for all sequential specs, all have just the 0#2020-02-1016:59ikitommialso, planning to support schamas as maps, via a helper.#2020-02-1017:00ikitommi[:vector {:min 0, :max 10} int?]
; =>
{:name :vector
:properties {:min 0, :max 10}
:children [int?]}#2020-02-1017:01ikitommithere is already a walker that recursively transforms between formats.#2020-02-1017:02ikitommiwill push these extras into malli.util, so there is just the absolute minimum stuff in the core.#2020-02-1021:47ikitommishould the m/name be m/type? In both React & json schema it's a type#2020-02-1021:48ikitommi[:vector {:min 0, :max 10} int?]
; =>
{:type :vector
:properties {:min 0, :max 10}
:children [int?]}#2020-02-1106:06ikitommi(mu/assoc-in nil [:a ::c :d] int?)
; [:map [:a [:map [:b [:map [:c [:map [:d int?]]]]]]]]
#2020-02-1106:08ikitommigreat thing about clojure is the functions compose nicely. HOFs are mostly 1:1 counterparts from Clojure (just cleaned up):
(defn update-in
"Like [[clojure.core/update-in]], but for LensSchemas."
[schema ks f & args]
(letfn [(up [s [k & ks] f args]
(assoc s k (if ks (up (get s k) ks f args)
(apply f (get s k) args))))]
(up schema ks f args)))#2020-02-1106:09ikitommi(defn update
"Like [[clojure.core/update]], but for LensSchemas."
[schema key f & args]
(let [schema (m/schema schema)]
(m/-set schema key (apply f (m/-get schema key nil) args))))#2020-02-1106:09ikitommiInternally, there are just -get and -set functions in a LensSchema protocol. Dead simple.#2020-02-1303:02eckardjfHow could I use a custom error message for a failed regex match? Something like:
(-> [:map [:a [#"^[0-9]+$" {:error/message "should match"}]]]
(mc/explain {:a "aaa"})
(me/humanize))#2020-02-1305:29eckardjfAh, I see what I was missing - this works
[:map [:a [:re {:error/message "should match"} #"^[0-9]+$"]]]#2020-02-1408:02ikitommiI'm bit surprised that the first one doesn't work.#2020-02-1414:20plexusI'm trying to use malli to coerce reitit query arguments, but not quite getting the results I'm expecting.#2020-02-1414:21plexusroute:
["/paginated/:table"
{:get {:coercion reitit.coercion.malli/coercion
:parameters {:path {:table string?}
:query {:page int?
:page-size int?}}
:responses {200 {:body any?}}
:handler (fn [{:keys [path-params params] :as req}]
{:status 200
:body "OK"})}}]#2020-02-1414:22plexusand router#2020-02-1414:22plexus(http/router routes {:data {:middleware [rrc/coerce-exceptions-middleware
rrc/coerce-request-middleware
rrc/coerce-response-middleware]}})
#2020-02-1414:22plexusis it reasonable to expect that this way a ?page=1 parameter will come in as {:query-params {"page" 1}}?#2020-02-1414:25ikitommiyes, but the coerced params are under :parameters, so_
{:parameters {:query {{:page 1}}}#2020-02-1414:22ikitommi{:table string?} --> [:map [:table string?]]#2020-02-1414:22plexusand not {"page" "1"}#2020-02-1414:23plexusahhh thanks, let me try that#2020-02-1414:23plexussame for :query then I suppose? [:map {:page ...}]#2020-02-1414:23ikitommididn’t add any sugar for the top-level maps in the malli coercion, so, all parameters need to be defined in the malli syntax.#2020-02-1414:24ikitommicrossed my mind to do that, would be 1:1 to switch the simple cases from data-specs to malli…#2020-02-1414:25plexusstill getting string...
:parameters {:path [:map
[:table string?]]
:query [:map
[:page int?]
[:page-size int?]]}
#2020-02-1414:26ikitommithere is an example app in https://github.com/metosin/reitit/tree/master/examples/ring-malli-swagger#2020-02-1414:27ikitommi["/math"
{:swagger {:tags ["math"]}}
["/plus"
{:get {:summary "plus with malli query parameters"
:parameters {:query [:map [:x int?] [:y int?]]}
:responses {200 {:body [:map [:total int?]]}}
:handler (fn [{{{:keys [x y]} :query} :parameters}]
{:status 200
:body {:total (+ x y)}})}
:post {:summary "plus with malli body parameters"
:parameters {:body [:map [:x int?] [:y int?]]}
:responses {200 {:body [:map [:total int?]]}}
:handler (fn [{{{:keys [x y]} :body} :parameters}]
{:status 200
:body {:total (+ x y)}})}}]]]#2020-02-1414:29plexussorry, had some connectivity issues there. Let me have a look at the example.#2020-02-1414:32plexusI guess you use all the reitit.*.middleware instead of ring-defaults now#2020-02-1414:45ikitommithere are useful middeware there, but for example the site is really heavy and many things are solved alread (like fallback to look up file resources)#2020-02-1414:45plexusok, progress. changed reitit.http/router to reitit.ring/router, and now I'm getting a 406 Not Acceptable#2020-02-1414:46ikitommithere is a bug somewhere in the malli-coercion, bumped into it few days ago, if the response is not valid, gives really weird error. will fix that soon.#2020-02-1414:47plexusadded Accept-Encoding=application/json and now it's a 405 Method Not Allowed#2020-02-1414:48plexusstill going to consider this progress 🙂#2020-02-1414:50ikitommiI have a minimal shadow-cljs + deps + reitit + malli example project, will good defaults, try to push that out before ClojureD#2020-02-1414:51plexusthat'd be great, thanks!#2020-02-1414:51plexuswas hoping not to manually have to parse those integers but maybe the trouble isn't worth it right now#2020-02-1414:51ikitommidefault reitit has no opinions, lot’s of things that need to decided, not the friendliest for new users#2020-02-1414:52ikitommiif you test the sample app I pointed and just strip away things?#2020-02-1414:52ikitommithere is also the request/response printer, just commented out.#2020-02-1414:52ikitommilet’s you see what the different mw do in the chain.#2020-02-1414:53ikitommiuses deep-diff btw 😉#2020-02-1414:55plexusnice, probably why it's been doing so well. most downloaded lambdaisland project 🙂#2020-02-1414:59plexuswe're working on clojurescript support for deep-diff BTW#2020-02-1421:36steveb8n@U07FP7QJ0 this is great news. when using Shadow-cljs and Cursive, there’s basically no easy visual diff tooling that I can find. cljs support in deep-diff will fill this gap perfectly#2020-02-1415:02plexusoh it's working now!#2020-02-1415:03plexusmisunderstood something about httpie :face_palm::skin-tone-2:#2020-02-1415:18ikitommiIt looks like malli need support for sequence schemas. I just need the simplest “varargs” case, why not do the whole thing while at it.#2020-02-1415:18ikitommiI’m thinking of re-using the map/multi syntax:
(m/valid?
[:cat
[:x int?]
[:y int?]
[:rest [:* string?]]]
[1 2 "kikka" "kukka"])
; => true
#2020-02-1415:19ikitommiif https://github.com/cgrand/seqexp would be ported to cljs, could use that behind the scenes.#2020-02-1415:21ikitommilike clojure.spec.alpha/conform, there could be something for that?
(m/destructure
[:cat
[:x int?]
[:y int?]
[:rest [:* string?]]]
[1 2 "kikka" "kukka"])
; {:x 1
; :y 2
; :rest ("kikka "kukka")}#2020-02-1415:22ikitommiwithout that, this would be just enough:
(m/valid?
[:cat int? int? [:* string?]]
[1 2 "kikka" "kukka"])
; => true
#2020-02-1415:23ikitommiideas most welcome on this.#2020-02-1415:31ikitommi(require '[net.cgrand.seqexp :as se])
(se/exec
(se/cat
(se/as :x int?)
(se/as :y int?)
(se/as :restz (se/* :string?)))
[1 2 "kikka" "kukka"])
;{:x (1)
; :y (2)
; :restz ("kikka" "kukka")
; :rest ()}#2020-02-1419:07borkdude@ikitommi fwiw, I ported clojure spec to a graalvm compatible subset here:
https://github.com/borkdude/spartan.spec#2020-02-1419:07borkdudeit contains all the regex ops#2020-02-1419:35ikitommioh, nice. Will have a look.#2020-02-1419:36borkdudeit's basically a copy with things removed#2020-02-1419:38ikitommiWhat if explainer would also report succesfully validated values in same format as errors? At top level, the m/explain it would return either errors or succefull values.#2020-02-1419:38ikitommiit would give destructuring basically for free.#2020-02-1419:41ikitomminow, only errors are pushed to the accumulator. It could be swapped Into a protocol, which has -explain-error and -explain-success functions. Schemas write to those and internally, in case of first error, it just tosses away the successes and collects only errors. In case of no errors, it would return the successes in the same explain -format!!#2020-02-1419:44ikitommias it contains both the paths to schemas and in data, one can "unexplain" in a generic way#2020-02-1419:57ikitommiSpec has conform, unform and explain , malli could only have explain for all these, but has validate for perf reasons. Still, less for more.#2020-02-1420:02borkdudein spec the bulk of the work is done in conform. this result is fed into explain which transforms that result to something readable, but even valid? uses conform#2020-02-1509:56ikitommiI don’t think result of s/conform is fed into explain. It either returns the conformed value of ::s/invalid. In case of latter, s/explain is called again with the original data. Which is fine, as it’s the failure path as you said.#2020-02-1509:57ikitommim/validate does much less than s/valid? and because of that - is much faster.#2020-02-1509:58ikitommior, actually, m/validator.#2020-02-1509:58ikitommi;; 40ns
(let [spec (s/and int? (s/or :pos-int pos-int? :neg-int neg-int?))
valid? (partial s/valid? spec)]
(cc/quick-bench
(valid? 0)))
;; 5ns
(let [valid? (m/validator [:and int? [:or pos-int? neg-int?]])]
(cc/quick-bench
(valid? 0)))#2020-02-1420:02borkdudeI think might have the philosophy that when something is valid it should go fast, but when something is invalid, it doesn't matter if you have to call conform again#2020-02-1420:03borkdudebecause something is wrong anyway#2020-02-1511:07ikitommino unform in spartan-spec?#2020-02-1512:41borkdudesince I've never used it, I didn't port it yet 😛#2020-02-1512:41borkdudePR welcome!#2020-02-1917:03ikitommi(require '[malli.core :as m])
(require '[malli.transform :as mt])
(def transformer
(mt/transformer
;; first run schema-based transformations named :before
{:name :before}
;; run json-things
(mt/json-transformer)
;; add default values
(mt/default-value-transformer)
;; run custom things
{:name :after}))
(m/decoder
[:map
[:name string?]
[:age int?]
[:address
[:map
[:street string?]
[:country [:enum "finland" "germany"]]]]]
transformer)
; => clojure.core$identity#2020-02-1917:04ikitommiI keep surprised how good the transformation engine really is.#2020-02-2414:01ScarHey everyone. Love using this tool.
Seems like validation works blazing fast, however mp/provide is slow.
for example - creating a schema from a vector of 200 items will take more than 2 minutes. The processing growth is linear so 20,000 would take about 9 hours :(.
I think it has to do with the fact the schema is built using exceptions.#2020-02-2414:03Scar#2020-02-2417:45ikitommioh, that's horrible. The only really perf optimized trails are -validator, -explainer and -transformer paths of schemas. Ideas welcome how to make the providers faster.#2020-02-2417:47ikitommithe algo is currently brute force, but I think if there were some kind of type-based narrower, it would not throw. e.g. Java Long would only be checked against predicates that work with that.#2020-02-2508:46Ben Slessoff the top of my head, wouldn't "errors as values" be faster? i.e. instead of throwing, return the data in a map with an :error key#2020-02-2515:39ikitommiErrors as values would be slower in the happy case / success - one would have to check always is the result an error or not. Try is basically zero cost in JVM, so the perf penalty (of throwing) only happens at the unhappy path.#2020-02-2515:41ikitommiI think the exceptions originate from the fact that the inferrer tries to make all registered schemas from it and they throw on invalid childs. E.g. for value 1, it tries to make a map with [:map 1], which throws on creation.#2020-02-2515:42ikitomminarrowing possible schemas to test would make it orders of magnitude faster, my guess on 2 (orders)#2020-02-2612:55eskosThere’s IIRC some newer optimizations in post-JDK9 versions for making stack&exception handling way faster, but I haven’t really looked into those beyond StackWalker…at least one of them was a throw-without-resolving-stack kind of operation; the stack unrolling is what generally is the slow part when exception gets thrown.#2020-02-2612:57eskosIt is also maybe possible to memoize the exception itself, although I wonder if that would work at all with the ex-info…#2020-02-2812:36ikitommijust in time for tomorrows talk 🙂#2020-02-2812:37ikitommithat’s clj-kondo talking#2020-02-2823:44JorinLooking forward!#2020-02-2908:55kszabogood luck with the talk!#2020-02-2911:27ikitommithanks! the slides are here: https://www.slideshare.net/mobile/metosin/malli-inside-datadriven-schemas#2020-03-0207:44eskosIs there going to be a video of this?#2020-02-2912:27ziltiNice, thanks! I enjoyed the presentation. Considering the library for the project at work now.#2020-02-2912:29ziltiYou mentioned something about generating input forms, right?#2020-03-0214:39ikitommiwe have been doing that is many projects, but with other schema-libs, nothing library-quality for malli yet. Did a spike using antd+malli, and learned that we need to add 1-2 extra args to m/-accept for this.#2020-03-0214:40ikitommionce it’s done, goal is to make an alternative schema backend for https://domino-clj.github.io/ from malli.#2020-03-0300:38ziltiI see. Thing is I am very interested in everything that gives me a bit of a hint for how to tacke flexible form generation for a project at work. Do you happen to have a link to something, no matter if done with another schema lib?#2020-02-2916:04Vincent CantinIs there some guidelines for contributing to Malli?#2020-03-0214:40ikitomminot yet, should be,.#2020-03-0214:40ikitomminot yet, should be,.#2020-03-0204:57Vincent Cantinif I understand correctly, the most natural way to implement the equivalent of spec/conform and spec/unform in Malli is to use a Transformer, isn't it?#2020-03-0214:42ikitommiare you interested in the branching information from s/conform?#2020-03-0214:43Vincent Cantinyes, totally. I need it for my Vrac project.#2020-03-0214:44ikitommiCurrently, there is no such thing. But one idea was that the m/explain also collected the explain info from the successfully validated values.#2020-03-0214:45ikitommi… that would contain all the needed branching information and could be used to produce “humanized” result like the one s/conform does.#2020-03-0214:46ikitommialso, would be easy to “unform” that as all the locations and values already exist there.#2020-03-0214:50Vincent CantinI am not totally convinced that the function explain should also be used for s/conform. The implementation might be similar, but explain and conform have different purposes.#2020-03-0214:51Vincent CantinThat’s why I was wondering if it would be better to have a separate function for that, maybe via a Transformer.#2020-03-0214:52Vincent CantinI did not finish to play with Malli, not sure if the Transformer pattern is suitable for those 2 functions.#2020-03-0214:55ikitommiyou should be able to do that with a transformer#2020-03-0209:32borkdudeEnjoyed the presentation, thanks!#2020-03-0210:24orestisI browsed the slides - is the Kondo integration a vision for the future or does it work today already? I’m super excited!#2020-03-0215:02ikitommiit was a 15min hack to check if that would work and it did! I just emitted a lint.edn which the fn description in the form clj-kondo understands it and hard-wired clj-kondo to read that file. I think there is betters ways to do this automatically. The m/defn is also not finished, but will be a 1:1 port from plumatic schema so cursive knows that too. Might take few (dev) days to get them cleaned up and pushed to master for testing.#2020-03-0218:00orestisSuper cool! I’ve wanted to generate specs for GraphQL input objects which are closed maps, and being able to statically analyze functions that consume them. Looks like this might be possible soon!#2020-03-0218:02orestisOh and I could probably also annotate React components that consume GraphQL...#2020-03-0213:03teodorluThanks for the great talk on ClojureD, @ikitommi!#2020-03-0213:05teodorluI stumbled over an article that I figured people here might find interesting:
QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs
The article discusses generating Clojure specs from OpenAPI specifications, and running generative testing against those. With Malli, I figure that process might become even simpler.
https://arxiv.org/pdf/1912.09686.pdf#2020-03-0214:56ikitommiConform/unform using explain would look something like this:
(def Schema
[:map
[:a [:or pos-int? string?]]
[:b string?]])
(m/explain Schema {:a 42, :"a"})
;{:schema [:map
; [:a [:or pos-int? string?]]
; [:b string?]]
; :value {:a 42, :"a"},
; :results [#Success{:path [1 1 1], :in [:a], :schema pos-int?, :value 42}
; #Success{:path [2 1], :in [:b], :schema string?, :value "a"}]}
(-> Schema
(m/explain {:a 42 :b "a"})
(me/humanize))
;{:a #Branch{1 42}
; :b "a"}#2020-03-0215:03Vincent Cantinin my use case with spec, I encountered a few times the need to apply custom transformations during s/conform , that’s also why I would think that a Transformer would be better if it can be easily customized to produce the format I want in one pass.#2020-03-0215:04ikitommi@teodorlu looks interesting. need to read that when extra time.#2020-03-0215:05ikitommiyeah, transformers give you the single pass. looking forward to seeing if that helps over spec.#2020-03-0217:18ikitommi@borkdude the :keys doesn’t work with :ret values with clj-kondo yet?#2020-03-0217:18ikitommion :args it seems to work, which is awesome 🙂#2020-03-0217:19borkdude@ikitommi what :ret type are you using?#2020-03-0217:21ikitommi{:args [{:op :keys
:req {:a {:op :keys
:req {:b {:op :keys
:req {:c :int}}}}}}]
:ret {:op :keys
:req {:b {:op :keys
:req {:c :int}}}}}#2020-03-0217:22borkdudeI don't think that's supported yet. As a fallback you can use :tag :map for now and post an issue about it.#2020-03-0217:22borkdudemaybe it works when you add :tag :map in that same map you pass to :ret now, so you can keep the code the same when it gets fixed?#2020-03-0217:25ikitommiwill write an issue, the :tag :map makes it a map, but the keys are not followed, at least (:a my-fn) succeed despite the ret says there is no such key#2020-03-0217:25borkdudethanks#2020-03-0217:59ikitommikinda bad explanation, but https://github.com/borkdude/clj-kondo/issues/783#2020-03-0415:56plexusI vaguely remember talk at ClojuTRE about generating forms / UI based on Malli. Has anyone tried something like that? Any interesting pointers?#2020-03-0416:53ziltiYea that is something I am highly interested in finding out#2020-03-0418:37ikitommigenerating UIs - there are many ways to do that:
1. walk the malli schema and emit an ui-application out of that, did a quick hack of that in a project, reagent + antd
2. walk the malli schema and emit an ui-schema out of that, to be rendered somehow
3. have both malli schema (“the model”) and an ui-schema (“the layout”) and use them together to generate the ui#2020-03-0418:39ikitommiin 1 & 2, the malli.visitor needs 1-2 extra args to be usefull: all walked schemas should know their place in the schema (e.g. the :in in explain) and in the data (the :path). This enables the ui-elements to be standalone, the ui-component knows where in the app-state the data needs to be stored.#2020-03-0418:40ikitommiI used formik-style custom helper to get the usability-stuff done right, dirty form state etc.#2020-03-0418:41ikitommibut, nothing Ready now, if someone has time and skills at making awesome forms, happy to help with that.#2020-03-0418:42ikitommithere is https://domino-clj.github.io/, idea was that it’s own model could be swapped with malli (just need a protocol to abstract the model and make malli-domino wrapper). It has also a graph-engine to calculate values & other cool features.#2020-03-0418:45ikitommichecked the js-side form generators last year, some good things to learn from there, e.g. https://github.com/rjsf-team/react-jsonschema-form & https://jsonforms.io/#2020-03-0418:51ikitommiin my ad-hoc generator, had ui-hints as properties like this: [:and {:ui/label "arvostelu", :ui/type :rate} int?]#2020-03-0418:52ikitommiemitting a https://ant.design/components/rate/#2020-03-0506:37ikitommihttps://github.com/metosin/malli/pull/184 <-- adding the :in to visitor, :path not needed.#2020-03-0506:38ikitommifxed :path with :multi schema: https://github.com/metosin/malli/pull/185#2020-03-0506:39ikitommi#2020-03-0506:40ikitommias with sequencial schemas, we don’t know the actual index of the value, so added just a placeholder “value here” using :malli.core/in marker. Comments welcome on that.#2020-03-0506:44ikitommianother options would be to:
1. use -1, e.g. temp-id like in datomic
2. use 0, the first index#2020-03-0507:57eskosre: Form generation based on malli discussed yesterday, I’ve been thinking that since malli has the capability, it could be interesting to provide the entire form with the validator functions from backend - in projects I’ve been eg. translations for error messages are managed on the backend services anyway, so generating a full-blown “form scheme” with not just what the desired data structure is but also how it’s validated and what the translations and help texts, labels etc. are in various languages and whatever else could actually be the most interesting usability wise.
This approach would of course mean doing some extra heavy lifting as binding this data would mean the UI component (think React) would need to have indirection to the path to data (both read and updates) instead of being instanced directly with the data, but that is a better architecture anyway. This was discussed in some conference last year, I’ll try to find a link…#2020-03-0508:21eskosFor the life of me I can’t find the post/talk. Anyway it was mostly just a suggestion that
1. have global state (eg. re-frame’s db)
2. all components should know only the path to their data in the state (`[:foo :went :to :bar]`)
3. all updates to data is done to that same path
What my caffeinated flow of mind above tried to convey was that with this pattern having the malli data to be mounted to some path in semi-generic fashion would allow instancing the form components quite nicely as well 🙂#2020-03-0508:24ikitommidid you check my code snipplet from above? allows to add the path to data…#2020-03-0508:25ikitommiui-components need to know how to handle sequences and :map-ofs, e.g. need to generate / retain indexes for the elementes. for nested maps & tuples, the path can be fully calculated forehand.#2020-03-0508:36eskosYes, now that I re-read that I think we’re at least in the same ballpark with how it should be.#2020-03-0508:37eskosgot me thinking, does malli handle seq keys for maps? 😅 there’s this one terrible API I saw recently which returned data like this {[:foo :bar] :baz [:more] :stuff}#2020-03-0508:52ikitommiwhat other ways there are to handle seq keys than to use seqs like in the example?#2020-03-0508:53eskosThe first step is to make sure that is even parsed correctly 🙂 afaik python/ruby/php can’t even read that in#2020-03-0508:54eskos(why am I on the weird side of this industry all the time…)#2020-03-0508:57eskosAnyway, the biggest concern with that is that correct validator is matched per key which actually a sequence; in my laziness I made that example uniform while in practice it probably wouldn’t be.#2020-03-0610:02teodorluJust pasted a huge JSON sequence into https://malli.io and got a nice, readable schema out, with optional values, sequences and everything in place.
What a pleasure! Thanks! 🙏#2020-03-0610:04mike_ananevHi! I'm trying to mu/merge two specs [:map ...] and [:multi ... ]. Why I always have only one of them in a result? Is there any examples?#2020-03-0610:08ikitommi@mike1452 they are of different types and the last one wins (like with clojure.core/merge)#2020-03-0610:09mike_ananev@ikitommi thanx.#2020-03-0610:09ikitommibut, the merge is far from perfect, not sure what happens if one merges two multis for example.#2020-03-0610:10ikitommii would assume that they get merged together, but not sure if the impl has a special condition just for :map (should be for MapSchemas, which both multi and map are)#2020-03-0610:11ikitommi… with that change, could actually merge those togerher. not sure if that’s a good idea thou :thinking_face:#2020-03-0610:11mike_ananevanother question: when I generate multi spec, why I have random values in dispatch type instead of those values that in multi spec?
(def mumap
[:multi {:dispatch :db/shared-channel}
["folder" [:map [:db/shared-channel string?] [:db/shared-channel-folder string?]]]
["web" [:map [:db/shared-channel string?] [:db/shared-channel-url string?]]]])
#2020-03-0610:12mike_ananevinstead of "folder" and "web" in shared channel i have random values. I expect only these two values#2020-03-0610:12ikitommithe generator is picked from the multi branch, which doesn’t limit the key.#2020-03-0610:12ikitommitry:
[:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]]
#2020-03-0610:14mike_ananevIt helps, thanks!#2020-03-0610:14ikitommias the dispatch can be anything, it’s non-trivial to use the branch name in a generator automatically#2020-03-0610:39mike_ananevI have data structure in which first half is always the same and second half is multi-spec based on :db/shared-channel value
(def sample1 {:db/type "h2",
:h2/user "sa",
:h2/filename "./boxdb",
:db/shared-channel "folder",
:db/shared-channel-folder "dev/resources/shared"})
(def sample2 {:db/type "h2",
:h2/user "sa",
:h2/filename "./boxdb",
:db/shared-channel "web",
:db/shared-channel-url ""})
#2020-03-0610:39mike_ananevI did multi spec
(def mumap
[:multi {:dispatch :db/shared-channel}
["folder" [:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]
[:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
[:h2/filename {:default "./boxdb"} string?]
[:h2/user {:default "sa"} string?]]]
["web" [:map [:db/shared-channel [:= "web"]] [:db/shared-channel-url string?]
[:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
[:h2/filename {:default "./boxdb"} string?]
[:h2/user {:default "sa"} string?]]]])
#2020-03-0610:40mike_ananevQuestion: how to avoid duplication of first half? (merge is not work like in spec1)#2020-03-0610:56mike_ananevI need something like:
[:map
[:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
[:h2/filename {:default "./boxdb"} string?]
[:h2/user {:default "sa"} string?]
[:multi {:dispatch :db/shared-channel}
["folder" [:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]]]
["web" [:map [:db/shared-channel [:= "web"]] [:db/shared-channel-web string?]]]]]
#2020-03-0610:59ikitommimany ways to do that, one would be to create a basemap, and merge that in all branches:
(def BaseMap
[:map
[:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
[:h2/filename {:default "./boxdb"} string?]
[:h2/user {:default "sa"} string?]])
(def Multi
[:multi {:dispatch :db/shared-channel}
["folder" (mu/merge BaseMap [:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]])]
["web" (mu/merge BaseMap [:map [:db/shared-channel [:= "web"]] [:db/shared-channel-web string?]]])]])
#2020-03-0610:59mike_ananev@ikitommi 👍#2020-03-0611:00ikitommithe oldest non-resolved issue in malli: https://github.com/metosin/malli/issues/14#2020-03-0611:01ikitommikinda like select in spec2.#2020-03-0611:17teodorluhttps://malli.io seems to fail to infer the schema of values containing instants. Example value:
{:some-date #inst "2020-02-02"}
Is this expected behavior?#2020-03-0611:22ikitommiI would not expect that. from a repl:
(require '[malli.provider :as mp])
(mp/provide [{:some-date #inst "2020-02-02"}])
; => [:map [:some-date inst?]]#2020-03-0611:30Vincent Cantin@ikitommi for https://github.com/metosin/malli/issues/180, do we want to also add support for :+ :* :? and some other operators? in the :cat driven sequence?#2020-03-0611:31Vincent CantinI hope to start this issue if I have some time this weekend.#2020-03-0611:32ikitommithat would be awesome!#2020-03-0611:32Vincent Cantin(if I have time)#2020-03-0611:33ikitommidid an intial spike, using the net.cgrand/seqexp , something like:#2020-03-0611:33ikitommi(def registry
(merge
default-registry
{:cat (cat-schema :cat)
:| (cat-schema :|)
:* (regex-schema :* se/*)
:*? (regex-schema : se/*?)
:+ (regex-schema :+ se/+)
:+? (regex-schema :+? se/+?)
:? (regex-schema :? se/?)
:?? (regex-schema :?? se/??)
:repeat (regex-schema :repeat? se/repeat)
:repeat? (regex-schema :repeat? se/repeat?)}))#2020-03-0611:33ikitommididn’t get very far thou.#2020-03-0611:33Vincent CantinCould you push this to a branch ?#2020-03-0611:38ikitommiactually, that’s about it. here are the codes I was playing with:
(require '[net.cgrand.seqexp :as se])
(se/exec
(se/cat
(se/as :x int?)
(se/as :y int?)
(se/as :rest2 (se/* string?)))
[1 2 "kikka" "kukka"])
; {:rest (), :match (1 2 "kikka" "kukka"), :x (1), :y (2), :rest2 ("kikka" "kukka")}
[:cat
[:x int?]
[:y int?]
[:rest [:* string?]]]
(se/exec-tree
(se/cat
(se/*
(se/as [:opts]
(se/cat
(se/as [:opts :x] string?)
(se/| (se/as [:opts :s] string?) (se/as [:opts :b] boolean?))))))
["-server" "foo" "-verbose" true "-user" "joe"])
(se/exec
(se/cat
(se/as :x (se/repeat 1 int?))
(se/as :y int?)
(se/as :restz (se/* string?)))
[1 2 "kikka" "kukka"])
; {:rest (), :match (1 2 "kikka" "kukka"), :x (1), :y (2)}
(se/*
(se/as [:sections]
(se/cat :h1
(se/as [:sections :ps]
(se/+ :p)))))#2020-03-0611:39Vincent Cantinthank you#2020-03-0611:39ikitommijust updated https://github.com/cgrand/seqexp/issues/11, which is about using seqexp here#2020-03-0611:39ikitomminot going to do anything for this for now, so all yours 🙂#2020-03-0611:39Vincent CantinI will start by writing tests#2020-03-0611:39Vincent Cantin.. then the implementation to make them pass#2020-03-0611:44ikitommi👍#2020-03-0611:34ikitommisure, doesn’t do anything thou#2020-03-0611:39borkdudewhy not just "steal" the impl from spec?#2020-03-0611:39teodorluUsing malli.provider/provide does the trick! Thanks.
I was surprised by what happened when I forgot to provide a sequence:
(mp/provide {:some-date #inst "2020-02-02"})
;; => [:vector some?]
com.github.metosin/malli {:git/url ""
:sha "9db4ff998b641d4a0cff4eb6d772fd0cb5d3b56c"}#2020-03-0712:25ikitommiMaps turn into sequences in Clojure, so it’s a sequence of a vector of keyword? and inst? , which both are some?
(seq {:some-date #inst "2020-02-02"})
;; => ([:some-date #inst"2020-02-02T00:00:00.000-00:00"])#2020-03-1011:55teodorluAh, that makes sense. Thanks.#2020-03-0611:41ikitommitaking the impl from spec is always an option.#2020-03-0611:42ikitommilast time I checked, the regexs where implemented using regular hashmaps in spec, would need at least some wrapping, don’t want to reserve maps for any one schema (family) type.#2020-03-0611:44ikitommigrand’s lib is 6y+ old, so “prior art” for this and all libs by Christophe have been great, also perf).#2020-03-0611:44borkdudeok#2020-03-0712:02ikitommiadded a contrinbution guide, 95% copy from reitit: https://github.com/metosin/malli/pull/186#2020-03-0812:06Vincent CantinRequest for comments on https://github.com/metosin/malli/pull/187#2020-03-0902:16Vincent Cantin@ikitommi https://github.com/cgrand/seqexp will need some adjustments if we want to use it in Malli:
• Its implementation is not cross platform.
• The maps returned by se/exec contain the keyed groups in the middle of :rest and :match , that could be a problem when the keys of those groups are controlled by Malli’s users.#2020-03-0906:16ikitommi@vincent.cantin ok. The cross-platform should be doable, ig Cristophe takes PRS (there should be an issue about that), not sure about the second. If that is not a good base lib to use, do you have plans on using something else? or from scratch?#2020-03-0906:33Vincent CantinI started to scratch my head about a new impl this morning. What's next depends on what features we want to offer the users.#2020-03-0906:35ikitommijust commented on the issue#2020-03-0906:37Vincent CantinI think that an optimized & naive impl might be faster on most use cases compared to a general algorith that works in theory faster and cover all edge cases.
I would like to put them to a benchmark on realworld usages.#2020-03-0908:26ikitommiWhat kind of edge cases would not be covered using a naive impl? Benchmarks most welcome!#2020-03-1002:51Vincent CantinAll the use cases would be covered by using a naive impl. Some edge cases might have a high computational cost in a naive impl while a smarter impl would mitigate the cost of their parsing.
The edge cases I am thinking about are consecutive sequences of variable length. But I am not sure if it is a real concern or my imagination.#2020-03-0906:48ikitommidid a quick draft of a swappable default registry: idea that the default registry could be bootstrapped into something that satisifes m/Registry protocol. There could be malli.mutable with an alternative impl using a extra atom to look for the schemas and malli.mutable/register!to register schemas into that. This would be a simple -> easy helper, already not happy with that. Comments welcome https://github.com/metosin/malli/pull/188#2020-03-0914:44ikitommi;; JVM started with -Dmalli.core.registry=malli.mutable/registry
(require '[malli.core :as m])
(require '[malli.mutable :as mm])
(mm/register! ::age [:and int? [:> 18]])
(m/validate ::age 20)
; => true
#2020-03-0914:45ikitommi🙈#2020-03-1002:33Vincent CantinI also don’t like the idea to lose the property of working with immutable values.#2020-03-1002:39Vincent CantinI would suggest users who don’t want to have to pass the registry as option to Malli all the time to just create their own wrapper functions.
;; in my-ns
(defn validate [schema data options]
(m/validate schema data (assoc options :registry my-registry)))
#2020-03-1002:42Vincent CantinI was facing the same situation in projects which use Vrac, and I used the wrapper pattern everywhere. It’s really not a big deal and it keep the satisfaction of working in an environment where we don’t even have to think about the side effects.#2020-03-0907:56eskosIs cross-platform regex even possible? Beyond basics I don’t think JS/Java/C# regex engines are even close to compatible with each other.#2020-03-0909:02Vincent Cantinwe are talking about validating data structures, not string content.#2020-03-0909:03eskos:thinking_face:#2020-03-0910:24ikitommi@U8SFC8HLP see https://clojure.org/guides/spec#_sequences#2020-03-0910:28eskosRight, thank you. I got confused, quite obviously 🙂#2020-03-0918:53shaunxcodeare recursive schemas possible w/ malli?#2020-03-0918:54shaunxcodee.g. defining a schema for a {:name "something" :kids [{:name "something" :kids [....]}]}#2020-03-0918:55shaunxcodeit does appear such things are possible in spec: https://gist.github.com/bhb/6cfcb3b38757442aec4ba5db46148699#2020-03-0919:08ikitommi@shaunxcode not yet, see https://github.com/metosin/malli/issues/20#2020-03-0919:11shaunxcodesorry just reading the whole of that linked issue and I see you guys are putting a lot of thought and effort into your approach so I will wait and see!#2020-03-0920:50ziltiI am about to start to try generating Malli schemas from PostgreSQL metadata. Very curious how far I can take this. If I find a way to store additional custom metadata in the database, I could end up generating 90% of a form simply by converting Postgres metadata, the rest being the layout.#2020-03-1108:13eskosJust curious, what kind of metadata you mean? PostgreSQL’s information schema is already quite extensive…#2020-03-1112:03bedersI have some code that does the same. I wasn't able to find a good solution for restricting string length. Maybe that has changed in the meantime.#2020-03-1113:06eskosIn PostgreSQL `TEXT` is string with infinite length (_=<https://wiki.postgresql.org/wiki/FAQ#What_is_the_maximum_size_for_a_row.2C_a_table.2C_and_a_database.3F|1GB> because that is any field’s maximum size_); string with specific maximum should be done with `VARCHAR` columns in which case `information_schema.columns` `character_maximum_length` has the column’s size limit information.#2020-03-1113:07eskosAssuming I’ve understood correctly, I seem to have a lousy track record on that part this week… 🙂#2020-03-1208:03viestiThe JDBC API also offers metadata, but probable less than direct PostgreSQL use. Prior art in spec land:
• https://github.com/tatut/specql
• https://github.com/viesti/table-spec#2020-03-1208:03viesti@U2APCNHCN the specql implementation might be really good help in looking what postgresql offers#2020-03-1208:58eskosJDBC https://docs.oracle.com/en/java/javase/11/docs/api/java.sql/java/sql/DatabaseMetaData.html is very extensive, haven’t tested but it seems with https://docs.oracle.com/en/java/javase/11/docs/api/java.sql/java/sql/DatabaseMetaData.html#getColumns(java.lang.String,java.lang.String,java.lang.String,java.lang.String) COLUMN_SIZE has the maximum length; the same TEXT/VARCHAR size detail as described above of course still applies.#2020-03-1216:09viestiyup, although I have a feeling that it might lack info on postgresql specific extensions#2020-03-1218:17ziltiYes, DatabaseMetaData is what I currently use. My showstopper right now is that I can't get the data transformations to work...#2020-03-1218:18ziltiI haven't heard of specql so far, but I will give it a look#2020-03-1307:39eskosMight be just me but if I were doing something very product specific, I’d steer far away from any kind of abstraction and just go as raw and bare in as possible… 🙂#2020-03-1310:25ziltiWell I think the problem wih going as raw and bare as possible is that you lose all the flexibility. And we're going to need that flexibility. My competition for creating forms are airtable and typeform. And I'd prefer we don't use either of those for our product#2020-03-1612:08teodorluQuickly model your tables with malli.provider/provide. Works well with JSON columns -- malli finds a specification that works with all your data.
(->> (with-open [conn (jdbc/get-connection db-datasource)]
(jdbc.sql/query conn ["select * from myschema.mytable"]))
(malli.provider/provide))
[:map
[:mytable/id int?]
[:mytable/name string?]]#2020-03-2010:36eskosGeneral consensus on whether this should work or not?
(require '[malli.util :as mu])
(mu/get-in [:map [keyword? int?]] [:a])
;=> returns nil, but I expected int?
I have nested maps with a few generic bits (my use case doesn’t allow composing schemas from reusable parts, yet) and noticed this behavior. Considering it’s supposed to mirror https://clojuredocs.org/clojure.core/get-in in functionality I’m hesitant to call this a bug, just wondering out loud… 🙂#2020-03-2010:53ikitommi(m/validate [:map [keyword? int?]] {:a 1})
; => false
(m/validate [:map [keyword? int?]] {keyword? 1})
; => true#2020-03-2010:53ikitommido you mean :map-of here?#2020-03-2011:22eskosHm, I think I broke something else in the process as well… a moment 🙂#2020-03-2011:23eskosYes indeed, :map-of#2020-03-2011:25eskosThis of course changes the nature of the error, although I’m still not willing to call this a bug; I’m kind of hoping for magic which obviously isn’t there… :)
=> (mu/get-in [:map-of keyword? int?] [:a])
Execution error (IllegalArgumentException) at malli.core/fn$G (core.cljc:26).
No implementation of method: :-get of protocol: #'malli.core/LensSchema found for class: malli.core$_map_of_schema$reify$reify__692
#2020-03-2013:01ikitommiwould you like to do a PR of making :map-of implement the LensSchema protocol?#2020-03-2014:54eskosSo had a quick stab at it, -get is literally just
(-get [_ key default] (if (key-valid? key) value-schema default))
but -set is a lot harder as semantically it doesn’t really make sense as the result should probably be some kind of weirdo map schema, eg.
(mu/assoc [:map-of keyword? int?] :foo boolean?)
;=> should probably create
[:or
[:map [:foo boolean?]]
[:map-of keyword? int?]]
and that wouldn’t work for subsequent calls, assoc-in etc…
One way to do this would be to allow :map to have default validators for kv pairs, effectively :map-of within :map, but that’s way too much work and decisions to do at this point of week 😅#2020-03-2108:39eskosComing back to this, would it make sense to have it as an option for open maps that one could specify default k/v validators for :map schema? This way the -get could work if the given validator matches and -set would switch the default k/v validator to whatever is given if it doesn't match any of the actual keys.#2020-03-2013:42eskosHmm, I could take a look at it at least. Haven’t delved into malli code base that much yet so no promises on that side but what the hey, it’s not like I’d have any other places to be… 😄#2020-03-2013:44ikitommimaybe we should do a form-heavy work project together and finalize all the things 😉#2020-03-2107:25Vincent Cantin@ikitommi Clojure spec’s cat is doing 2 things at the same time and I don’t like that:
• It says something about the container type of the sequence,
• It says something about the content of the sequence.
In Malli, I propose to move the description on the type of container to a separate structure schema name and to remove this responsibility from :cat and :alt .
Maybe we can have something like:
[:cat [:a int?] [:b :string?]] ;; matches '(1 "a") and [1 "a"]
[:in-list [:cat [:a int?] [:b :string?]]] ;; matches '(1 "a")
[:in-vector [:cat [:a int?] [:b :string?]]] ;; matches '[1 "a"]
#2020-03-2605:57ikitommi@U8MJBRSR5 I guess you can say already [:and vector? [:cat ...]]#2020-03-2107:34Vincent Cantinor we can call them seq-of list-of and vector-of#2020-03-2107:44Vincent CantinI am preparing a commit in my MR, on which we can comment later.#2020-03-2205:59Vincent Cantin@ikitommi What is the reason Malli use [:map [:x int?] [:y int?]] instead of [:map :x int? :y int?] ?#2020-03-2206:00Vincent Cantin(I am asking for :map because I am wondering the same for :cat and :alt )#2020-03-2206:32mike_ananevI guess that inside [:x int?] you can set many things: customized error message, validation function etc. Without vector[:x int?] expression structure becomes unreadable.#2020-03-2206:55Vincent CantinGood point.#2020-03-2207:12Vincent CantinI updated the tests for :cat and :alt . Those tests represent the proposal I make for the data format which constitutes the API for this part of Malli.
Feedback welcome ! https://github.com/metosin/malli/pull/187/files#2020-03-2414:53naomarikThere a way to use a custom :error/message for :or?
Something like this:
[:or
{:error/message "nil, string, or int"}
nil?
string?
int?]#2020-03-2606:01ikitommi@naomarik currently, no. And same applies for adding error messages on map entries. Would we solved with some way to figure out parents: https://github.com/metosin/malli/issues/120#2020-03-2606:01ikitommiideas welcome on how to do that.#2020-03-2606:03ikitommithought of copying props to childs from map entries and ors, but that would duplicate the info in m/form etc. knowing the parent at runtime would be cleanest way to resolve these.#2020-03-2712:56ikitommiin a project swapping plumatic into malli and I think there should be more built-in schemas for common thigns, strings and dates. Maybe something like:
[:string {:min 1, :max 10}]
[:int {:min 1, :max 10}]
[:date {:min "2012-04-23"}]
[:date-time {:min "2012-04-23 00:00:00", :format "yyyy-MM-dd HH:mm:ss"}] #2020-03-2713:10eskosSounds good to me. Maybe some of those should be aliases for other more in-depth schemas, but it’ll help the users a lot if there’s a good collection of those available out of the box.#2020-03-2713:19eskos…huh, even? and odd? are not in the predicate registry. I was about to mention how to something like [:range {:from 0 :to 10 :step 2}] would really be just [:and [:int {:min 0 :max 10}] [:even?]] and thought to check that. Sorry, my mind’s wandering 🙂#2020-03-2713:28eskosAlso pos? neg? seem to be missing. Are these left out on purpose or just forgotten?#2020-03-2714:15ikitommithe initial set of predicates for taken from clojure.spec, those which have generators defined. no other reason, could be added.#2020-03-2714:16eskosRighto 🙂#2020-03-3009:13teodorluI just had a pleasant experience during my team's standup. I'm working on code that depends on some PostgreSQL tables with JSON columns. We're migrating to a new set of tables given new needs. In that change, we're also moving some data from within the old JSON structure out into normal table columns.
To give insight into my work, I generated schemas for the JSON with malli.provider/provide, and shared the generated malli schemas with my team. Explaining malli took 30 seconds and one example:
"Malli is a schema system for JSON."
"This JSON:"
{"x": 1,
"name": "something"}
"Gets this schema:"
[:map
[:x int?]
[:name string?]]
This resulted in an experienced team member saying "oh, that somePropertyName, I didn't even know we had those."
Felt great! 🎉#2020-03-3013:23pithylessIf I want to add a custom tag to the registry, to support something like [:string {:min 1, :max 10}] - do I need to create a brand new IntoSchema (e.g. modeled similarly to m/fn-schema on -leaf-schema but with support for properties) + define a custom mg/-generator ? Or is there some way I can reuse more of the existing malli machinery?#2020-03-3017:55ikitommi@pithyless there are not many helpers in the core currently, but reifying the IntoSchema is a good way to do those. I think all the separate concerns (JSONSchema, Generators, Providers etc) should have a supporting protocol so one could easily satisfy all the concerns with one reify-block. Currently requires protocol + set of multimethods. For core stuff, it’s intentional to have the mm’s: the core remains small as it doesn’t have to know much/anythning about the other concerns.#2020-03-3017:56ikitommialso, there is a PR for discussion about swapping the default registry - to support spec-like mutable registries.#2020-03-3017:57ikitommiI think if there would be a set of type-kind-of schemas in malli core, one would not need the current core predicates at all. e.g. :string instead of string? , :date-time instead of inst etc.#2020-03-3110:00eskosI’ve been lately thinking whether malli should adopt git’s porcelain vs plumbing mentality for the schemas, but haven’t drafted anything sensible yet about it. To me there seems to be a lot of cases where the more complex/abstract schema really is just a composition of simpler ones, eg. range is just int with min and max and stepping validation (`[:range {:from 0 :to 10 :step 2}) => [:and [:int {:min 0 :max 10}] [:fn '(fn [i] (= 0 (mod i 2))]]`) or [:map-of keyword? string?] could be [:map {:default-kv [keyword? string?]}] <- this one doesn’t exist! just an example) etc. but this would require quite a bit of thought on choosing what the base schemas in registries are and what counts as plumbing and what doesn’t… 🙂#2020-03-3113:46teodorluVega/Vega Lite does something similar. Vega Lite is high-level data format (for plotting), and compiles down to Vega.
https://vega.github.io/vega-lite/#2020-04-0111:34mike_ananevWhat is an idiomatic way to put docstring into spec? via properties?#2020-04-0112:50teodorluI suspect that if you want to use Malli for your specs, you might want to model your documentation as pure data too. I'm eager to hear where other people would prefer to store those docstrings.#2020-04-0113:04ikitommi@mike1452 keys :title and :description are the default keys for docs, but free to use any other keys too.#2020-04-0113:04ikitommithose keys get pulled into JSON Schema & Swagger Schema docs.#2020-04-0113:04ikitommi(and should be used for other documentation too)#2020-04-0214:34ikitommifixed a bug on collection type transformations, which turned maps into vectors. Will push a new SNAPSHOT to clojars as need it with reitit (which uses leiningen)#2020-04-0214:34ikitommihttps://github.com/metosin/malli/pull/192#2020-04-0216:55dcjPresumably mailli schema properties are open?
So I can specify a (new) key, that I use for my own purposes?
Example: assign :postgres/type "some-postgres-type", later I walk schema and "make it so" via connection to database?#2020-04-0220:07dcjApparently so, a test schema for a database table:
(def metadata-schema
[:map {:postgres/schema "slm"
:postgres/table "measurement_metadatas"}
[:id {:postgres/type :bigserial
:postgres/key :primary} int?]
[:instrument-id {:postgres/type :bigint
:postgres/null? false} int?]
[:datafile-id {:postgres/type :bigint
:postgres/null? false} int?]
[:seq {:postgres/type :integer
:postgres/null? false} int?]
[:origin {:postgres/type :float8
:postgres/null? false} double?]
[:scale {:postgres/type :float4
:postgres/null? false} float?]
[:starting-timestamp {:postgres/type :timestamptz
:postgres/null? false} inst?] ;; TODO: fix
[:sample-rate {:postgres/type :float4
:postgres/null? false} float?]
[:log-interval {:postgres/type :float4
:postgres/null? false} float?]
[:weighting {:postgres/type :text
:postgres/null? false} string?]
[:manifest {:postgres/type :int
:postgres/null? false} int?]
[:time-zone {:postgres/type :int
:postgres/null? false} int?]])
And, as a quick experiment, wrote fn to print DDL text driven by the above, yielding:
CREATE TABLE slm.measurement_metadatas (
id bigserial PRIMARY KEY,
instrument_id bigint NOT NULL,
datafile_id bigint NOT NULL,
seq integer NOT NULL,
origin float8 NOT NULL,
scale float4 NOT NULL,
starting_timestamp timestamptz NOT NULL,
sample_rate float4 NOT NULL,
log_interval float4 NOT NULL,
weighting text NOT NULL,
manifest int NOT NULL,
time_zone int NOT NULL
);
#2020-04-0220:12dcjIs there (should there be) a long? predicate in malli?#2020-04-0300:37dcjOK, I have been yak shaving experimenting with Malli most of the day, having lots of fun!
I have a bunch of questions:
• After creating a new predicate, I want to assoc it into the existing predicate-registry map (creating my own). -register-var and -register are private. Obviously I can copy/paste these functions...suggestions?
• Given my metadata-schema above, I want to transform a map via this schema. I get how m/decode is helpful, and I can specify key-transformer but I'm scratching my head about why/how I need to specify specific value transformers eg, string-transformer . Given that my schema has specified the type of each value (eg. int? ), then what I want to say is "coerce whatever you see into the specified type" I get that not all coercions can be pre-defined, and maybe there needs to be a way to add coercions....advice?
• Again, given my metadata-schema above, let's say I want to obtain all the keys of that map schema. The top level form is a schema, and m/children gives me the list of children, but I can't figure out how to obtain the "name" of each child, without resorting to digging into the current implementation of that vector. In other words, sure I can first each child, but is there some better way to get that info/field?
• Similar to the above, say I want to get each key in my map schema transformed in some way. When I want to transform a map, I do this: (m/decode my-schema my-map (mt/key-transformer {:decode csk/->snake_case_keyword}) because SQL column names are snake_case, but wouldn't it be better to include that transformer into my schema itself? Maybe I should define a new property at the map level :postgres/key-transformer and I (-> my-schema m/properties :postgres/key-transformer)? Thoughts?
• Similar to the schema navigation/accessor questions above. when I first attempted to write the function to walk the schema and generate the DDL based on the :postgres/ properties, I tried to use m/accept with an function that would look at each [schema properties _] it encountered, but this didn't work because AFAICT the children are not schemas themselves. Finally I gave up and walked the schema using the obvious vector destructuring, but I don't feel this the right way to do it...#2020-04-0305:34ikitommi• properties are open, there is a bunch of reserved keys (`:title` , :description, :default, :min, :max etc.) and some namespaces that are reserved for extensions (`json-schema`, gen, swagger, encode, decode etc.). Should be documented and I guess a good convention could be “new unqualified might be used by malli in the future, to be safe, use qualified keys to avoid future clashes”. Goal is to describe the properties with malli schemas to get good error reporting on those too#2020-04-0305:37ikitommi• -registerand -register-var could be made non-private, just undocumented. But in the end, it’s just assoc into the map. There is https://github.com/metosin/malli/pull/188 for discussion whether to support mutable registries oob#2020-04-0305:40ikitommi• coercions, sorry, didn’t get the question.#2020-04-0305:42ikitommi• all :map schemas implement MapSchema , which allows you to say (m/map-entries schema) to get the entries out#2020-04-0305:49ikitommi• if you want to transform just the entities, you could create a new Transformer, that only mount to :maps which have the :postgres/schema defined, this way, it doesn’t effect all maps. You can add a custom encoder & decoder just like with key-transformer. #2020-04-0305:56ikitommi• the m/accept… each IntoSchema is responsible for it’s own syntax so walking over schemas requires some knowledge of the schema in question. With maps, children is the sequence of entries, which are tuple3 of key-props-child.#2020-04-0306:00ikitommiYou can run (m/accept Schema m/map-syntax-visitor) to see the structure to be walked. With Java, the Visitor impl would have dispatch methods with different class signatures, with clojure, one can use multimethods for doing schema-based dispatch, see malli.generator or malli.json-schema for examples.#2020-04-0306:01ikitommiCurrently, only :map and :multi use custom syntax for it’s contents, later some variants of :or , :cat and :alt too (to support named branches). Could be others.#2020-04-0306:02ikitommiThere is also m/schema-visitor helper for walking.#2020-04-0306:03ikitommihope this helps @dcj#2020-04-0306:05ikitommithe long? predicate… could be, or :long type schema to companion :date-time, :date etc#2020-04-0306:05ikitommi(there is no long? in clojure core, for some reason)#2020-04-0306:07mike_ananev@ikitommi any plans to release the malli? malli is awesome! without release I clone malli to my projects every month manually.#2020-04-0306:21dcj@ikitommi Thank you for all the answers/advice/tips, I will try those tomorrow!
I will also try to better explain my coercion question, but too tired to do it justice ATM...#2020-04-0307:46ikitommi@mike1452 not going to do that yet, but soon. with deps, you can depend on th the latest commit directly. With Leiningen, there is [metosin/malli "0.0.1-20200305.102752-13"], could start pushing new SNAPSHOTS after each merge.#2020-04-0307:48ikitommias soon as malli works fully with reitit, could push first alpha. some small hiccups still.#2020-04-0308:16eskosedit: all of this is garbage 🙂 see below
`long?` predicate might be missing from Clojure due to the original design decision of having transparently coercing number types based on number size (#2020-04-0308:22eskosint? is true for Bytes, Shorts, Integers, Longs, while integer? is true for Integers, Longs, Clojure BigInts, BigIntegers, Shorts and Bytes. I don’t think there’s specifically benefit for checking if value is strictly Long, but YMMV.#2020-04-0309:42teodorluDifferentiating between Integer and Long would allow generating precise Java source code from a schema#2020-04-0317:29dcj@ikitommi m/map-entries and map-schema-entry structure as 3tuple of [key props child]is very helpful.
For some reason, it makes me uneasy to just destructure the map-schema-entry 3tuple (instead of having accessor functions),
but the good news is that you enforce that format, so even if the entry was specified without props, you provide the correct 3tuple, so destructuring works well!
How would I test for a map schema? Is there a better way than: (= :map (m/name my-schema)) ?#2020-04-0317:34dcj@ikitommi (m/accept Schema m/map-syntax-visitor) is an awesome way to understand how the walking works, thanks!
ATM, for my current "database table schema" task, I am pretty happy with m/map-entriesand processing the returned map-schema-entries....
But I'm sure I'll want to walk schemas at some point and good to know this.#2020-04-0317:47dcj@ikitommi I don't want to mutate the default registry, I just want to add a few more things to the default registry.
My initial feeling is that I am happy to provide my registry as options map.
Maybe there should be public functions to support users adding to the registry, to make their own variants
I'll just copy/paste from -register and -register-var for now....#2020-04-0318:34dcj@ikitommi WRT my earlier coercion question(s), I studied the README and transform.cljc further,
looks like this machinery will do everything I want, I'll just need to provide/add my own encoding and decoding functions...I'll get to that soon enough#2020-04-0318:47dcjRegarding long? and/or :long This requires further thought. If I am specifying a key :epoch-millisconds then it feels like that should specified as a long, not an int, so transformers can provide the correct type.
Currently:
(def +string-decoders+
(merge
+json-decoders+
{'integer? string->long
'int? string->long
So, ATM those transforms do produce longs, but this seems somewhat arbitrary and not as precise as it might be.
If the schema doesn't care about the the specific numeric type, then specifying something that includes a variety of types is fine.#2020-04-0323:39dcjMore progress this afternoon:
• Was able to add a new predicate to the default-registry, creating my-registry, and used it to define and validate a schema
• Created a value transformer, and was able to get it invoked to transform a value via decode.
Questions:
• What is the meaning/use of the :name key in the mt/transformer input map? Can I put anything I want here? Should I use a namespaced keyword?
• What other uses are there for the (optional) options map argument? (other than :registry)?#2020-04-0401:18dcjSo when I create a registry, and use it with m/schema, does that registry get saved with the schema (in -options)?#2020-04-0403:05dcjWRT:
What is the meaning/use of the :name key in the mt/transformer input map?
Is it the case that executing mt/transformer {:name :foo :decode ... :encode...} registers that transformer via the key :foo somewhere, and it can be retrieved via mt/transformer {:name :foo}) ?#2020-04-0403:12dcjDoes a transformer that uses {:compile have to be defined inline within a specific schema, or can it be a "general" transformer that gets called against any schema ?
The README example shows `
{:compile '(fn [schema _]
What is the second arg to that function (ignored here) ?#2020-04-0406:31ikitommi• :name in transformer is used for property-based transformation, I think only unqualified work right now (could be changed to support qualified too), having a transformer named :postgres allows one to define :encode/postgres and :decode/postgres keys in schema properties
• currently no other used built-in keys than :registry in options, you can add you own - options are passed in mostly all user-defined callback functions. You should namespace your own option keys, just like custom schema property values, to avoid clashes
• Schemas should save the options they were created with. I believe all in-built schemas do that.
• :compile - it’s a feature of the interceptor, so can be used for general stuff, see the default-transformer (adding default values to schemas if missing): https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L315-L341
• second arg is the options#2020-04-0406:34ikitommiI would like to see to options be One place to customize how malli works: defining the registry, adding error messages & localizations (https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L5-L55), generarators etc, whether to use serializable functions or not (sci adds some to the bundle size in cljs) etc.#2020-04-0406:35ikitommihaving a malli schema to define the core malli options would also give good errors on invalid options 😉#2020-04-0406:37ikitommi@dcj here’s the line: only non-qualified transformer names are supported right now https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L60. Issue & PR welcome if you want qualified too.#2020-04-0406:40ikitommican you ask for the entries: (satisfies? malli.core/MapSchema schema)#2020-04-0406:41ikitommilooking at the source:
(defn map-entries
"Returns a sequence of 3-element map-entry tuples of type `key ?properties schema`"
([?schema]
(map-entries ?schema nil))
([?schema options]
(if-let [schema (schema ?schema options)]
(if (satisfies? MapSchema schema)
(-map-entries schema)))))#2020-04-0406:41ikitommiso, return nil or entries.#2020-04-0406:43dcj@ikitommi no current need for qualified transformer names.
Thank you again for all these answers.
I will need to experiment/study transformers more tomorrow...
#2020-04-0406:57dcj@ikitommi Can :decode/foo keys in schema props be on the props for a single key in map, or only at the top :map level?#2020-04-0406:58ikitommiall schemas, but not currently the map-entries, later, see https://github.com/metosin/malli/issues/86#2020-04-0406:59ikitommibut this works:
[:map
[:x [int? {:decode/postgre ...}]]]#2020-04-0407:00ikitommithe parent-child thing (and support for recursive schemas) is the last big thing to resolve before a release. Ideas welcome on that.#2020-04-0412:44pithylessWhat kind of naming conventions have people found useful for malli-specs? In clojure.spec, one has a separate registry of names (s/def ::user) but malli just uses existing ns vars. Some idioms I've seen: user-schema, User, ns.just.specs/user ; a plain (def user) I'm afraid will clash too frequently with other/local bindings in your domain.#2020-04-0416:29dcj@pithyless Never having used malli 'in anger', my plan is to create a new/separate project/repo definitions to hold schemas and the like for a collection of interrelated projects.
There will be subdirectories within definitions for specific domains, each with its own schema file/ns. So: definitions.foo.schema
Within each schema file/ns I will just def the indvidual schemas, eg (def measurement (m/schema ...)
I'll require these as needed like this: [definitions.foo.schema :as foo.schema]
So then I'll access as foo.schema/measurement
Disclaimer: Currently re-writing some code using my third schema tool, third database access library, and third time library, so clearly I can't get anything right the first N times 🙂#2020-04-0419:20dcjI'm having trouble groking property-based transformation.
Here is a simplified example:
(def my-schema
(m/schema
[:map {:closed true
:encode/postgres csk/->kebab-case-keyword
:decode/postgres csk/->snake_case_keyword}
[:id {:optional true
:postgres/type :bigserial
:postgres/key :primary} int?]
[:instrument-id {:postgres/type :bigint
:postgres/null? false} int?]
]
))
(def sample
{:id 12345
:instrument-id 67890})
(m/decode my-schema sample (mt/key-transformer {:name :postgres}))
;; {:id 12345, :instrument-id 67890}
I was expecting the :instrument-id key to be transformed to :instrument_id
I have tried quoting the values of :encode/postgres and :decode/postgres various ways both single-quote and hash-single-quite to no avail.
Also tried to use mt/transformer different problems.
How should I do this?#2020-04-0420:21dcjI guess my example above is confused about the encode/decode directions/terminology, be that as it may, I still can't seem to get the key-transformers to run, no matter what....#2020-04-0510:04ikitommi@dcj the key-transformer doesn’t understand the :name. It’s simple transformes all keys, not usefull in your case. You should use normal transformer instead. I created a gist, which might show the needed core for the thing you are looking for, using both named property-driven and a custom transformer options. Hope this helps: https://gist.github.com/ikitommi/5d71fc6b320e996b1ac9c17c4d412b71#2020-04-0510:13ikitommi@pithyless coming (and loving) Plumatic Schema, I have use the class-like naming: User, Address. Currently porting a Schema-project to use Malli, so the var names existed already. If I would add the schemas to the registry, might want to use qualified keywords: (assoc registry ::user User).#2020-04-0510:14ikitommiCurrently, haven’t added any domain schemas into the registry. is anyone doing that?#2020-04-0510:29kwrooijenI was thinking of doing this. I want to create UI elements based on the type of the schema. My initial thought was to use properties, but creating a registry could also be a valid option. But I'm still learning all the ins and outs of Malli though.#2020-04-0512:46kwrooijenIs there an idiomatic way of writing documentation for keys? I can't seem to find anything about that in the README. Currently I have this, which works, but feels very hacky.#2020-04-0513:17ikitommijust for keys:
(def Human
[:map
[:human/name {:description "Human name"} string?]
[:human/age {:description "Human age"} int?]])
(doseq [[k p _] (m/map-entries Human)]
(println k "->" (-> p :description)))
;:human/name -> Human name
;:human/age -> Human age#2020-04-0513:17ikitommifor values:
(def Human
[:map
[:human/name [:and {:description "Human name"} string?]]
[:human/age [:and {:description "Human age"} int?]]])
(doseq [[k _ s] (m/map-entries Human)]
(println k "->" (-> s m/properties :description)))
;:human/name -> Human name
;:human/age -> Human age#2020-04-0513:18ikitommior:
(def Human
[:map
[:human/name [string? {:description "Human name"}]]
[:human/age [int? {:description "Human age"}]]])
(doseq [[k _ s] (m/map-entries Human)]
(println k "->" (-> s m/properties :description)))
;:human/name -> Human name
;:human/age -> Human age#2020-04-0513:22ikitommias Slack history loses all code, wrote a gist: https://gist.github.com/ikitommi/7c26aebf2b0f58d58432c884c8a55b5a#2020-04-0513:23kwrooijenAhh nice, m/map-entries is what I needed simple_smile also didn't know the :and was redundant. There are a couple of example that use :and + opts in the README#2020-04-0513:24kwrooijenSo would the convention be :description then? Since that was also used by json_schema ?#2020-04-0513:27ikitommiyes, :title and :description should be used, I’ll add those to README.#2020-04-0513:27ikitommi(malli.json-schema/transform
[:map
[:human/name [string? {:description "Human name"}]]
[:human/age [int? {:description "Human age"}]]])
;{:type "object",
; :properties #:human{:name {:type "string", :description "Human name"},
; :age {:type "integer", :format "int64", :description "Human age"}},
; :required [:human/name :human/age]}#2020-04-0513:28kwrooijenOk, thank you for the clarification 👍#2020-04-0516:09sudakatuxHi#2020-04-0516:30sudakatuxSo how can i test malli on a lein project#2020-04-0516:30sudakatux?#2020-04-0516:35dcj[metosin/malli "0.0.1-SNAPSHOT"]#2020-04-0516:35sudakatuxcool just found it at the same time. Thank u#2020-04-0516:37dcj@ikitommi Thank you for the gist, that is super helpful!
You provided two different examples, named property-driven transformations and named transformations, ATM I think I like the named transformations more, but will mull it over...
The named transformations version of postgres-transformer answered further questions I had about how to use :complle
Would it be a good/interesting idea to store transformers like these in the options of the schema?
If so, then presumably they could be accessed later when needed for encode/decode calls....#2020-04-0600:16dcjRegarding map schemas:
To get all the map keys from the schema itself: (mapv first (m/map-entries my-schema))
So far there have been two different cases where I need to get all the "encoded" keys from the schema itself, is there any way to do that?
For the moment, I have defined the "key-encode-fn` and I store that away in the schema, then when I need the encoded keys, I go fetch it, and (mapv (comp key-encode-fn first) (m/map-entries my-schema)
And when I am creating the encoders for a key related mt/transformer, again, I go get this function, and (partial -transform-keys key-encode-fn)
This all works if I define and follow the convention above.
Is there another/better way to get the "encoded" keys from a schema?#2020-04-0600:28dcjFor the record, defining transformers with :compile can be difficult to get correct, until you get an accurate mental model of when in time the various parts really happen on the 'def time' vs 'execution time' continuum. I struggled with this for a while, then eventually went back to a working example @ikitommi gave me, and put printlns at various places.
That really helped! Finally I got it working! Very useful and powerful!#2020-04-0601:17dcjA map-entry is a 3tuple vector containing [`key ?properties schema]` , and as data/text, the properties is/are optional...
If you've been following along with my "map-schema with support for Postgres tables" example(s), one map entry might like like this:
[:id {:optional true
:postgres/type :column
:postgres/datatype :bigserial
:postgres/key :primary} int?]
Functionally, this is working great.
From a schema readability perspective, IMHO, there is quite a lot of "textual distance" between the key, and the schema, when properties are specified/used,
I really need to search for the schema of this key...
So far, the schemas I have specified for keys are textually simple/short (mostly the built-in ones, or similar predicates I have created)
Given that malli is "Pre-alpha, in design and prototyping phase" I wonder if switching the order of the propertes and the schema in a map-entry would improve readability/comprehension...
Disclaimers: maybe this is a terrible idea. I have only a small number of hours of experience with malli, and only with pretty simple map schemas. I imagine this change would be extremely painful for everyone involved, and that could easily outweigh any benefits (assuming there are any benefits). I will happily live with the current map entry structure...#2020-04-0607:35kwrooijenAn alternative is adding :id to the registry as {::id int?}. and then using the upcoming map syntax (if this pr gets approved https://github.com/metosin/malli/pull/194)
(def my-registry {::id int?})
(m/schema
[::id {:optional true
:postgres/type :column
:postgres/datatype :bigserial
:postgres/key :primary}]
{:registry my-registry})
Something like this. And I think it will be easier to extend the registry later (if I'm reading the issues / PRs correctly)#2020-04-0607:42kwrooijenAlso I think the reason for this format: [key ?properties schema]
is because it's inline with the Hiccup syntax https://github.com/weavejester/hiccup
Meaning that it would be a bit couter-intuitive to change the order. I could be wrong though.#2020-04-0615:27dcjSeems like it depends on how ‘terminal’ the schema element in a map-entry is.
If the schema element was going to describe a another map/list/etc, then switching the props and schema would be horrible, which is probably why the current ordering is the way it is....
#2020-04-0616:01dcjWhich is the case with the example schema grounded_sage showed in a subsequent post.
So now I see why the current ordering is best.
Never mind....#2020-04-0616:04dcjNEVERMIND, I now see why current element ordering is best....
A map-entry is a 3tuple vector containing [`key ?properties schema]` , and as data/text, the properties is/are optional...
If you've been following along with my "map-schema with support for Postgres tables" example(s), one map entry might like like this:
[:id {:optional true
:postgres/type :column
:postgres/datatype :bigserial
:postgres/key :primary} int?]
Functionally, this is working great.
From a schema readability perspective, IMHO, there is quite a lot of "textual distance" between the key, and the schema, when properties are specified/used,
I really need to search for the schema of this key...
So far, the schemas I have specified for keys are textually simple/short (mostly the built-in ones, or similar predicates I have created)
Given that malli is "Pre-alpha, in design and prototyping phase" I wonder if switching the order of the propertes and the schema in a map-entry would improve readability/comprehension...
Disclaimers: maybe this is a terrible idea. I have only a small number of hours of experience with malli, and only with pretty simple map schemas. I imagine this change would be extremely painful for everyone involved, and that could easily outweigh any benefits (assuming there are any benefits). I will happily live with the current map entry structure...#2020-04-0615:54grounded_sageI’m having trouble updating a schema. Not sure how to do it.
Current schema
[:map
[:Data
[:map
[:CommunicationChannels
[:vector
[:map
[:Identifier [:map [:Value string?] [:Details [:map [:ID string?] [:SystemID string?]]]]]
[:ChannelType [:map [:Type int?] [:Name string?]]]
[:Content string?]]]]]]]
Desired schema ->
[:map
[:Data
[:map
[:CommunicationChannels
[:vector
[:map
[:Identifier [:map [:Value string?] [:Details [:map [:ID string?] [:SystemID string?]]]]]
[:ChannelType [:map [:Type int?]
[:or
[:Name "EMail"]
[:Name "Mobile"]
[:Name "Fax"]]]]
[:Content string?]]]]]]]
#2020-04-0616:21grounded_sageOh figured it out. Have to use the :enum#2020-04-0620:40kwrooijenI'm trying to run the malli tests locally, but getting the following error:
Exception: clojure.lang.ExceptionInfo: Kaocha ClojureScript client failed connecting back.#2020-04-0620:40kwrooijenThis doesn't happen in CI though. And I do have some tests that are actually failing, but this error is swallowing them all 😕#2020-04-0620:40kwrooijenDoes anyone else have this issue?#2020-04-0620:41kwrooijenMacOS Catalina, by the way#2020-04-0702:48grounded_sageI can’t get Union working on my schemas#2020-04-0702:54grounded_sageThis is the gist with the schemas in it.#2020-04-0702:56grounded_sagehttps://gist.github.com/groundedSAGE/4752ff810afe184ce119416a76eb8894#2020-04-0708:13eskos@grounded_sage it seems neither of those are valid schemas, I split the first one and reduced it into this which fails
user=> (m/validator [:map
#_=> [:Data
#_=> [:map
#_=> [:GroupMemberships [:vector [:or]]]]]])
Execution error (ExceptionInfo) at malli.core/fail! (core.cljc:73).
:malli.core/no-children
#2020-04-0708:14eskosThat :or seems lingering, typo?#2020-04-0708:15grounded_sageYea I thought that was strange. Can malli generate invalid schemas?#2020-04-0708:15grounded_sageBecause this was after doing schema inference.#2020-04-0708:17eskosIf that is a result of inferring then I guess the answer is yes 🙂 If you can figure out the minimal set of input data which produces invalid inferred schemas, I’m sure an issue is warmly welcomed.#2020-04-0709:39ikitommiSounds like a bug.#2020-04-0813:31grounded_sageJust posted an issue for it.
https://github.com/metosin/malli/issues/196#2020-04-0712:55sudakatuxHi#2020-04-0721:22kwrooijenAnyone now if this is possible? :spritesheet/frame and :spritesheet/animation are technically both optional. But the map MUST have one or the other. This code snippet is invalid, the :or doesn't work (disclaimer, I'm working off of this PR https://github.com/metosin/malli/pull/194 which is why I can use this qualified-keyword syntax)#2020-04-0813:27aisamuThe way I've done such xor-like unions in regular spec is to move the or up one level:
[:or
[:map
:spritesheet/name
:spritesheet/frame]
[:map
:spritesheet/name
:spritesheet/animation]]
#2020-04-0723:16dcjLet's say I have a value that is an enum:
(def Weighting
[:enum :dB-A :dB-C :dB-Z])
I can specify that as a value in another schema:
(def MyMap
(m/schema
[:map
[:weighting Weighting]]))
Is there a way to turn the value schema into a "type" that malli understands?
Let's say I need to transform the numbers 0,1,2 into the enumerated values...
(defn weighting->kw
[n]
(if (int? n)
(case n
0 :dB-C
1 :dB-A
2 :dB-Z
n)
n))
I specify a value transformer as:
(mt/transformer
{:name :weighting
:decoders {:enum weighting->kw}})
But in writing this, I am reaching into Weighting to pull out the :enum
If I had two different enum values then dispatching via "value is an enum" doesn't seem like it will work, they will both resolve to enum.
How does one handle this?
Seems like I would want to specify the decoder as {:Weighting weighting->kw}#2020-04-0813:27aisamuThe way I've done such xor-like unions in regular spec is to move the or up one level:
[:or
[:map
:spritesheet/name
:spritesheet/frame]
[:map
:spritesheet/name
:spritesheet/animation]]
#2020-04-0804:34ikitommiping @kevin.van.rooijen#2020-04-0804:36ikitommiI’m not a fond of too much extra syntax for :map. If the spec-like :or or :and are liked, I guess there could be :keys schema for that?#2020-04-0807:45kwrooijenThanks! This does exactly what I need. One question for clarification though; When talking about too much extra syntax, do you mean :or / :and or are you referring to my pull request?#2020-04-0807:46ikitommijust referring to the :or and :and , like spec1 has.#2020-04-0807:46ikitommiwill check the PRs as soon as have time. busy times.#2020-04-0807:49kwrooijenAhh ok thanks. Take your time!#2020-04-0812:32yonatanelI need something similar and even a bit more complex. I want to require a value either at top level or nested:
;Valid:
{:key1 "yo"}
;Also valid
{:nested {:key2 "yo"}}
It’s not the same key but they have the same meaning and one must exist.
Currently I have this schema:
[:or
[:map
[:key1 any?]
[:nested {:optional true} [:map [:key2 {:optional true} any?]]]]
[:map
[:key1 {:optional true} any?]
[:nested [:map [:key2 any?]]]]]
but I have more keys like this, and some options can be either of 3 keys so I need 3 maps for each key and then combine all of it with :and.#2020-04-0804:42ikitommi@dcj easiest way to attach a decoder just for that enum would be:
(def Weighting
[:enum {:decode/enum weighting->kw} :dB-A :dB-C :dB-Z])
and then applying it with:
(m/decode MyMap {:weighting 0} (mt/transformer {:name :enum}))
; => {:weighting :dB-C}#2020-04-0804:45ikitommiif you want to have a transformer for that, you can do the same as I gisted about the :postgres/table => add some hint to the enum and create a transformer that attaches in :compile just for that schema. hope this helps.#2020-04-0813:32kwrooijenYeah that's a good option as well. But that will probably lead to a lot of duplication (since my example was just a stripped down version)#2020-04-0814:18aisamuYup, it was quite annoying#2020-04-1010:19yonatanel@kevin.van.rooijen meanwhile I have this require-either function that adds :fn schema to the map with a humanized error message and makes the relevant keys optional (https://gist.github.com/yonatane/7bd1d38449f56b3aea62eb391964aee4)#2020-04-1119:32dcjJust to confirm: When specifying keys in a map schema, it is fine to http://use.so a namespaced keyword, eg: [:instrument/id int?] AFAICT from one quick test, this "just works". The schema can be defined, and validate works as expected... Correct?#2020-04-1120:22pithylessYes. There's even an open documentation issue for this - https://github.com/metosin/malli/issues/189#issue-590324344#2020-04-1220:02sudakatuxHi#2020-04-1220:03sudakatuxSo im testing malli with cohersion and swagger. I was wondering when following the example online. the type dispatcher for swagger. its an empty object. how does one pass the descriminator value#2020-04-1220:04sudakatuxSo multi here https://malli.io/?value=%5B%7B%3Atype%20%3Asized%2C%20%3Asize%2010%7D%0A%20%7B%3Atype%20%3Ahuman%2C%20%3Aname%20%22tiina%22%2C%20%3Aaddress%20%7B%3Astreet%20%22kikka%22%7D%7D%5D&schema=%5B%3Avector%0A%20%5B%3Amulti%0A%20%20%7B%3Adispatch%20%3Atype%7D%0A%20%20%5B%3Asized%20%5B%3Amap%20%5B%3Atype%20%5B%3A%3D%20%3Asized%5D%5D%20%5B%3Asize%20int%3F%5D%5D%5D%0A%20%20%5B%3Ahuman%0A%20%20%20%5B%3Amap%0A%20%20%20%20%5B%3Atype%20%5B%3A%3D%20%3Ahuman%5D%5D%0A%20%20%20%20%5B%3Aname%20string%3F%5D%0A%20%20%20%20%5B%3Aaddress%20%5B%3Amap%20%5B%3Astreet%20string%3F%5D%5D%5D%5D%5D%5D%5D#2020-04-1220:04sudakatuxthe type attribute. say i want to pass that value. how should i pass it#2020-04-1220:05sudakatuxbecause in swagger its shown like :properties {:type {},#2020-04-1307:37ikitommimaybe :swagger/discriminator :type?#2020-04-1307:38ikitommihaven’t used the discriminator, so not sure how it should work. what is the expected swagger schema?#2020-04-1310:45sudakatuxso i would expect i could "type":"mytype"#2020-04-1310:45sudakatuxGM#2020-04-1310:50sudakatuxSo my probelm was that i could not POST since it could not figure out the type#2020-04-1310:50sudakatuxthe way i got what i wanted was by doing this#2020-04-1310:51sudakatux(m/validate schema (m/decode
schema data
(mt/transformer mt/strip-extra-keys-transformer mt/string-transformer)))))#2020-04-1310:51sudakatuxwhere schema is#2020-04-1310:51sudakatux[:multi
{:dispatch :type, :decode/string (fn* [p1__40150#] (update p1__40150# :type keyword))}
[:other [:map [:data [:map [:nombre string?] [:apellido string?] [:edad int?]]] [:type [:= :other]]]]
[:one [:map [:data [:map [:nombre string?] [:apellido string?] [:kk int?]]] [:type [:= :one]]]]]#2020-04-1310:52sudakatuxand data#2020-04-1310:53sudakatux{:data {:nombre "nombre" :apellido "apellido" :edad 12} :type "other"}#2020-04-1310:54sudakatuxon an ideal I would have liked to be able to use that schema in the {:paremeters {:body here}}#2020-04-1313:01plexusam I right that this is not supported? should it be?
(def m [:map
[:i {:error/message "not a number"} int?]])
(me/humanize (m/explain m {:i "123"}))
;; => {:i ["should be int"]}#2020-04-1313:02plexusI find in general there's some annoying asymmetry between properties on map entries vs properties schemas. See also my PR for adding :gen/gen / :gen/elements / :gen/fmap support to map entries#2020-04-1313:07sudakatuxsorry deleted the message by mistake#2020-04-1313:07sudakatuxwill take a look#2020-04-1313:08plexusthanks @U8JTE9PG8. I'm interested in general if users should expect props on schemas to also work on map entries, or if that's not a design goal.#2020-04-1516:26ikitommiThanks @U07FP7QJ0 for the PR! Will check it soon along others. I would like to have first class entry meta, not sure what is the best way for it.#2020-04-1313:05plexusI suppose you can do this
(def m [:map
[:i
[int? {:error/message "not a number"}]]])#2020-04-1513:44orestisI saw the ClojureD talk where it’s mention that there was some experiments tying malli and clj-kondo together. Is that code somewhere available? I’d love to have compile-time checks of some map usage. I don’t care particularly about the syntax right now, mostly about the hooking up kondo and its analyzer to validate malli schemas.#2020-04-1516:22ikitommi@orestis it was a quick hack, not published to repo. The m/defn is 1:1 from plumatic schema, emitting a clj-kondo file into the project. No generic conversion and plumatic port just a poc. Would take a day to make it real.#2020-04-1516:28orestisSo essentially it was just emitting clj-kondo specific schemas for the function, right? As if you hand-typed the kondo conf manually. I can work with that...#2020-04-1516:32ikitommiyes. Generic malli Schema -> clj-kondo schema transformer would be great - and relatively easy to do, just like the json-schema transformer.#2020-04-1516:29ikitommi@jstuartmilne if you know the form of the swagger schema, you could add the swagger transformation for :multi. It's an multimethod doing that, so can be done in user space. PR welcome when it works ;)#2020-04-1516:30ikitommiAlso, you can write the swagger manually, just add a :swagger key to schema and a full schema as value. Overrides everything#2020-04-1622:51sudakatuxcool thanks. it working how i want now 🙂#2020-04-1609:41grounded_sageThis bug https://github.com/metosin/malli/issues/196 is pretty critical for us atm. I’m happy to try jump in and fix if someone can point me in the right direction with some ideas on how to fix it.#2020-04-1612:02ikitommi@grounded_sage I think the culprit is here: https://github.com/metosin/malli/blob/master/src/malli/provider.cljc#L41#2020-04-1612:03ikitommifor empty sequences, it tries to figure out what is a valid schema and finds empty :or#2020-04-1612:05ikitommi(require '[malli.provider])
(malli.provider/provide [[]])
; => [:vector [:or]]
#2020-04-1612:25ikitommi@grounded_sage should be fixed in master.#2020-04-1614:22grounded_sageLegend! #2020-04-1612:27ikitommi(provide [])
; => any?
(provide [[]])
; => [:vector any?]
(provide [[[[[[[]]]]]]])
; => [:vector [:vector [:vector [:vector [:vector [:vector any?]]]]]]
#2020-04-1614:25grounded_sageThis is so helpful for the work I am doing atm where we are ingesting messy JSON from third parties. I use Malli to show me all the variations of schemas their JSON can have. Check the frequencies to see which ones we want to support. Can deep diff the schemas to see the differences. Then use the generated unified schema as the base for robust Meander pattern matching. #2020-04-1616:31ikitommiSounds cool! do you write the meander definitions manually or programmatically from schemas?#2020-04-1616:40grounded_sageCreate them manually. Would be interesting to see if I can generate a nice starting point from a Malli schema.. haven’t dove in that much yet to see if it’s practical.#2020-04-1616:43grounded_sageReminds me I did find another bug or unexpected behaviour earlier. Where doing unions over a number of schemas resulted in created some nested schema definitions that weren’t there. I got around this by just capturing the original JSON for the various schemas#2020-04-1920:39teodorluAre anyone using Malli for recursive schemas? Attached schema generated with malli.provider/provide.
In spec, I'd do something like this (not tested, and missing some properties):
(s/def ::title string?)
(s/def ::children (s/coll-of ::node))
(s/def ::node
(s/keys :req-un [::title
::children]))#2020-04-1921:30roklenarcicI am trying to add an error message to an existing schema, I tried simply saying: [:and {:error/message "X"} myschema] and that didn’t work#2020-04-2006:10eskos:and requires two leaves, you have only one.#2020-04-2006:28eskos@ikitommi What would be your preferred media for discussing some design choices malli could potentially have a stake in? I’ve been thinking for quite a while that there’s two topics I’d like to have persistently discussed, decided and documented, namely the plumbing&porcelain concept for schemas themselves I’ve mentioned before and swagger2 vs openapi3 vs postman collections generation, and while we could discuss those things here, the ephemeral nature of Slack and this unpaid Clojurians Slack instance especially isn’t very good documentation wise… 🙂#2020-04-2014:10aisamuI'll just chime in to mention that some folks here (#data-science IIRC) have been using Zulip for that with apparent success!
Doesn't lose history and has a much better (i.e. usable) threading model#2020-04-2015:22teodorluSecond chime in: I've been using Zulip quite a bit, and I think I prefer it to Slack. I don't see a huge difference, but it would be possible to have a single malli channel with different topics, where design could be one.#2020-04-2015:25aisamuYup!
I actually consume this slack's content through Zulip's mirror.#2020-04-2015:25teodorluHuh, cool. Didn't know that was possible. Do you happen to have a link?#2020-04-2015:30aisamuSure!
https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/malli
Each one of slack-archive's topics is a channel (after you invite @UFNE73UF4, that is)#2020-04-2015:41eskosLooks like Zulip’s free tier has limited search, are you sure it’s not going to lose history?#2020-04-2015:41eskosOr is someone paying/hosting that? 🙂 Sorry, not familiar with Zulip’s hosting scheme…#2020-04-2015:44aisamu> Zulip Cloud Standard is free for open source projects!
🎉#2020-04-2106:11eskos@ikitommi that could maybe actually work for Metosin’s OSS? --^#2020-04-2110:30eval2020Zulip-admin here. We're indeed using Zulip's OSS-plan (https://www.zulipchat.com/for/open-source/)
Happy to answer any more questions.#2020-04-2110:31eval2020btw I see @UFNE73UF4 is no longer a member of this channel... :thinking_face:#2020-04-2110:58eval2020scratch that (bots don’t appear in the member-list)#2020-04-2114:03teodorluThanks for keeping Zulip working 🙂 It's been really helpful for doing Scicloj-work 💯#2020-04-2114:38eval2020yw!, great to hear.#2020-04-2208:49plexuslate to the thread here but I'd like to propose ClojureVerse for this. Long form asynchronous is more suitable for writing out proposals and discussing their merit. Happy to set up a Malli category.#2020-04-2212:21aisamuWorth noting that Zulip also favours long form, async content (with proper quoting, nesting, etc) given the threading model.
(And I also consume ClojureVerse through Zulip's mirror - annoucements, but still) simple_smile#2020-04-2310:05ikitommioh, what would the category mean? all the discussion still go to the main feed?#2020-04-2310:08ikitommiBut, as the discussions are anyway in threads in clojureverse, one could just not read the malli thread/s if that’s not relevant to them.#2020-04-2311:03teodorlu@ikitommi
Here are all the project threads: https://clojureverse.org/c/projects/24
And here's the shadow-CLJS category:
https://clojureverse.org/c/projects/shadow-cljs/20#2020-04-2006:41ikitommi@teodorlu recursion schemas, discussion here: https://github.com/metosin/malli/issues/20#2020-04-2007:27teodorluThanks for the reference!#2020-04-2006:48ikitommi@roklenarcic @suomi.esko there are two ways that the properties are used:
1. top-down: generators use this. the first generator that is found is used. [:and {:gen/elements ["a" "b" "c"], :error/message "fail"} string?] short-circuits on that gen. decoders & encoders compose top-down, json-schema overrides work this way etc.
2. after-the-fact: error messages are looked for after the error has happened. here: it the example, the failure happens at string? , which doesn’t know any of the parent decorations for errors.
this is bad, and should be fixed in a coherent way, related issues: https://github.com/metosin/malli/issues/120 & https://github.com/metosin/malli/issues/86#2020-04-2006:50ikitommithe error message thing could be “easily” fixed so that the error message formatter looks also at the parent schemas (all the information is available in malli.core/explain result.#2020-04-2006:51ikitommi… but would not solve [:map [:x {:error/message "fail"} string?]]], where the info is in the entry meta.#2020-04-2006:51ikitommiideas welcome, will put my hammock on the backyard this week, so will think about how to do this#2020-04-2006:56ikitommi@suomi.esko maybe a open malli video chat? google hangouts for example later this week? issues have persistent discussion, but as things link, the discussion is bit scattered.#2020-04-2007:13eskosHmm, video chat could probably be an interesting way to solve this, as I have no idea if this is like 5, 15 or 50 minute topic and it’s not quite clear to myself either entirely what the whole scope is…#2020-04-2014:10aisamuI'll just chime in to mention that some folks here (#data-science IIRC) have been using Zulip for that with apparent success!
Doesn't lose history and has a much better (i.e. usable) threading model#2020-04-2020:51ikitommi(def Country
[:map {:id "Country"}
[:name [:enum :FI :PO]]
[:neighbors any?]])
(def Burger
[:map {:id "Burger"}
[:name string?]
[:description {:optional true} string?]
[:origin [:maybe [:maybe Country]]]
[:price pos-int?]])
(def OrderLine
[:map {:id "OrderLine"}
[:burger Burger]
[:amount int?]])
(def Order
[:map {:id "Order"}
[:lines [:vector OrderLine]]
[:delivery [:map
[:delivered boolean?]
[:address [:map
[:street string?]
[:zip int?]
[:country Country]]]]]])#2020-04-2020:52ikitommi(require '[malli.mermaid :as mermaid])
(mermaid/class-diagram Order)
;"classDiagram
; class Country {
; + :name [:enum :FI :PO]
; + :neighbors any?
; }
; class Burger {
; + :name string?
; + :description string?
; + :origin Country
; + :price pos-int?
; }
; Burger o-- Country
; class OrderLine {
; + :burger Burger
; + :amount int?
; }
; OrderLine o-- Burger
; class OrderDeliveryAddress {
; <<embedded>>
; + :street string?
; + :zip int?
; + :country Country
; }
; OrderDeliveryAddress o-- Country
; class OrderDelivery {
; <<embedded>>
; + :delivered boolean?
; + :address OrderDeliveryAddress
; }
; OrderDelivery *-- OrderDeliveryAddress
; class Order {
; + :lines OrderLine
; + :delivery OrderDelivery
; }
; Order o-- OrderLine
; Order *-- OrderDelivery"#2020-04-2020:53ikitommi#2020-04-2020:53ikitommihttps://github.com/metosin/malli/pull/198#2020-04-2020:54ikitommi(59 loc 😉)#2020-04-2022:00kwrooijenWow that's awesome#2020-04-2022:04kwrooijenThe tree structure logic is the mermaid namespace, right? I think it could be useful to have generic function to generate a dependency tree#2020-04-2208:49plexuslate to the thread here but I'd like to propose ClojureVerse for this. Long form asynchronous is more suitable for writing out proposals and discussing their merit. Happy to set up a Malli category.#2020-04-2212:21aisamuWorth noting that Zulip also favours long form, async content (with proper quoting, nesting, etc) given the threading model.
(And I also consume ClojureVerse through Zulip's mirror - annoucements, but still) simple_smile#2020-04-2105:18ikitomminow in malli.mermaid, but could push some of the building blocks into malli.util:
• function to resolve names for the anonymous nested schemas
• function to pull out relations between schemas that match a precidate#2020-04-2105:20ikitommithe actual mermaid code would be ~20 lines. would be easy to add a DOT printer too.#2020-04-2105:21ikitommimermaid doesn’t seem to allow special symbols in the class names, would have liked Order$Delivery$Address instead for the derived class names, easier not to clash. DOT allows that.#2020-04-2121:00geraldodevHi , I was trying to create a malli zipper and was using children to get down the tree. It happens that it crashes when there is no children. Is there a function for that ?#2020-04-2121:03geraldodevhttps://gist.github.com/geraldodev/f637da76bdac78fe8f7e330db43fe311
A malli zipper if anyone want to criticize is more than welcome#2020-04-2523:03sudakatuxHi#2020-04-2523:04sudakatuxi have a question regarding the json-schema suport#2020-04-2523:04sudakatuxSo if i go to the example:#2020-04-2523:04sudakatux[:map
{:title "Address"}
[:id string?]
[:tags [:set keyword?]]
[:address
[:map {:json-schema/title "Something"}
[:street {:json-schema/title "Something Else"} string?]
[:city string?]
[:zip int?]
[:lonlat [:tuple double? double?]]]]]#2020-04-2523:09sudakatuxand add title to :street that will not work i would expect to see "Something Else" as a title property for street#2020-04-2523:09sudakatuxI do see "Something" as the title for Addess#2020-04-2523:09sudakatuxGenerated JSON-schema :#2020-04-2523:10sudakatux{:type "object",
:properties {:id {:type "string"},
:tags {:type "array",
:items {:type "string"},
:uniqueItems true},
:address {:type "object",
:properties {:street {:type "string"},
:city {:type "string"},
:zip {:type "integer",
:format "int64"},
:lonlat {:type "array",
:items [{:type "number"}
{:type "number"}],
:additionalItems false}},
:required [:street :city :zip :lonlat],
:title "Something"}},
:required [:id :tags :address],
:title "Address"}#2020-04-2523:11sudakatuxWhy is not "Something Else" a title for street am i missing something?#2020-04-2815:23ikitommi@jstuartmilne it’s a known issue, to be solved, see https://github.com/metosin/malli/issues/86#2020-04-2815:23ikitommifor now, this works: [:street [string? {:json-schema/title "Something Else"}]]#2020-04-2815:24ikitommie.g. put the meta into entry schema, not to entry itself.#2020-04-2815:24ikitommithis needs a day in the hammock to figure out how to best resolve that.#2020-04-2815:25ikitommi(good thing I just found my hammock from the basement :)), also ideas & help welcome.#2020-04-2815:30sudakatuxahhh cool#2020-04-2815:30sudakatuxwill try that#2020-04-2815:30sudakatuxthank u verry much#2020-04-2915:47daveliepmannHow much control does Malli currently give over map keys? For instance that they should be of a particular set, or within a range, or keywords or strings, but without specifying the key precisely? I have an existing non-relational database for which I'd like to infer schemas with mp/provide , but it seems to get confused around hashmaps that are built to store user input (e.g. a mapping from username to info).#2020-04-2916:18ikitommi@daveliepmann there is :map-of , like in spec.#2020-04-2916:20ikitommithe mp/provide is not smart enough yet to figure those out. Could be..#2020-04-2916:23daveliepmannThanks!#2020-04-2920:27teodorlu@daveliepmann @ikitommi I was also getting more corarse-grained results with provide than I would like. I would love to be able to provide a "starting point" for the schema. Please excuse my backseat driving:#2020-04-3008:36ikitommi@teodorlu Agree, would be useful. I think someone could make a thesis (and read other people theses ;)) out of the providers. The current impl is really naive (69 loc), help most welcome. Ambrose might have some math for this already on typed clojure#2020-04-3008:36ikitommi#2020-04-3008:36ikitommi#2020-04-3008:38teodorluI'll have a look at the provider source 🙂#2020-04-3014:39daveliepmann@teodorlu I had a similar thought under which your hypothetical looked like [:map [:strength int?] #{"mage" "paladin" "ranger"}] in my head 🙂#2020-04-3014:46daveliepmannThe shortest real example I have is [:map-of #{<domain-specific items>} [:map-of #{<more domain-specific items>} boolean?]] . (The first set of domain items is limited but long enough to make me not want to enumerate each key to the same spec of values; the second set of domain items is IDs from another system that can't be enumerated in a spec but that I might want to check the existence of.) On some of the larger schemas I can see some difficulty getting the API for mp/provide to play nicely with a skeleton schema, because of how nested it would have to be.#2020-05-0123:49geraldodev(m/decode int? "23.abc" mt/string-transformer) gives me a string on clojure on cljs it gives me an 23 integer. it strips the integer part and passes. is it normal ?#2020-05-0411:47katoxHi all, I neeed a simple clojure.test check whether a struct conforms to Malli schema. I came up with this helper:#2020-05-0411:47katoxhttps://gist.github.com/katox/d5817209aec78edb052f851c86e34f99#2020-05-2412:42rutledgepaulvfyi I'm working on something similar to https://github.com/rjsf-team/react-jsonschema-form for malli using semantic-ui-react components w/ reagent and it's coming together nicely so far. good job tommi and other contributors 🙂 . i'll share something as i make more progress (just started yesterday)#2020-05-2508:29ikitommilooking forward to this!#2020-05-2508:42ikitommicould someone check the reasoning in https://github.com/metosin/malli/pull/201?#2020-05-2508:45ikitommiI now understand why schema.core/both got deprecated. :or is kinda complex when you wan’t to cover both validation and transformation. Same thing was “fixed” in spec-tools some weeks ago: results of a transformed branch needs to be validated to know if the branch was good.#2020-05-2811:12Vincent CantinI faced the same issue in my lib, and tried to separate shape-type elements from the predicates which are checking properties on them.#2020-05-2508:46ikitommischema had s/conditional, maybe Malli should have :conditional too :thinking_face:#2020-05-2508:48ikitommicould be implemented using :multi with just few extra lines to swap the dispatch function & values.#2020-05-2614:54sudakatuxHi#2020-05-2615:03sudakatuxim looking for a way to filter out#2020-05-2615:04sudakatuxrecursively in a nested schema every schema containing a certian property#2020-05-2615:04sudakatux`
[:identity [:map [:user map?] [:profile map?] [:token [string? {:deleteMe true}] ]]]]
#2020-05-2615:05sudakatux(my-cool-filter [:identity [:map [:user map?] [:profile map?] [:token [string? {:deleteMe true}] ]]]])#2020-05-2615:05sudakatux[:identity [:map [:user map?] [:profile map?]]]]
#2020-05-2615:06sudakatuxI suppose i should be looking at the visitor. since i can see it runns through every node. but im not sure how to filter it out#2020-05-2805:26ikitommi@jstuartmilne something like:
(m/accept
[:map
[:user map?]
[:profile map?]
[:nested [:map [:x [:tuple {:deleteMe true} string? string?]]]]
[:token [string? {:deleteMe true}]]]
(fn [schema children _ options]
(if-not (:deleteMe (m/properties schema))
(let [children (filterv (if (m/map-entries schema) last identity) children)]
(m/into-schema (m/name schema) (m/properties schema) children options)))))
; => [:map [:user map?] [:profile map?] [:nested :map]]#2020-05-2805:27ikitommie.g. check if the schema has the property, remove if it has. if not, you have to remove the nil childs as it’s currently not valid syntax. There are two child syntaxes: the normal and the (map)entry-one. need to handled differently.#2020-05-2805:28ikitommithat doesn’t drop the empty nodes, e.g. :map without any keys is valid, same with empty :tuple etc.#2020-05-2810:09ikitommianyone interested in doing malli.xml? something like:
(def User [:map [:users [:vector [:map [:user string?]]]]])
(m/decode User "<users><user>simo</user></users>" malli.xml/xml-transformer)
; => {:users [{:user "simo"}]}
(m/encode User {:users [{:user "simo"}]} malli.xml/xml-transformer)
; => "<users><user>simo</user></users>"#2020-05-2810:09ikitommirelated: https://github.com/metosin/muuntaja/pull/114#2020-05-2812:06sudakatuxThank u so much.Ive been struggling witth that#2020-05-3013:07ikitommiConverting between vector (hiccup) and map-syntax:
(defn ->map-syntax
([?schema] (->map-syntax ?schema nil))
([?schema options] (m/accept ?schema m/map-syntax-visitor options)))
(defn <-map-syntax
([m] (<-map-syntax m nil))
([{:keys [name properties children]} options]
(let [<-child (if (-> children first vector?) (fn [f] #(update % 2 f)) identity)]
(m/into-schema name properties (mapv (<-child #(<-map-syntax % options)) children)))))
=>
(def Schema
[:map
[:id string?]
[:tags [:set keyword?]]
[:address
[:vector
[:map
[:street string?]
[:lonlat [:tuple double? double?]]]]]])
(->map-syntax Schema)
;{:name :map,
; :children [[:id nil {:name string?}]
; [:tags nil {:name :set
; :children [{:name keyword?}]}]
; [:address nil {:name :vector,
; :children [{:name :map,
; :children [[:street nil {:name string?}]
; [:lonlat nil {:name :tuple
; :children [{:name double?} {:name double?}]}]]}]}]]}
(-> Schema
(->map-syntax)
(<-map-syntax))
;[:map
; [:id string?]
; [:tags [:set keyword?]]
; [:address
; [:vector
; [:map
; [:street string?]
; [:lonlat [:tuple double? double?]]]]]]#2020-05-3013:18ikitommiShould the support for map-syntax be baked into malli.coredirectly? One could create a schema with m/schema using either one (can’t mix - the whole Schema tree needs to be created with same syntax - not to lose any extensibility) and the created Schema would know the syntax it’s created with: m/form would return in the original one.#2020-05-3013:24ikitommihmm. maybe not now (thinking aloud), have some really big map schemas in a project with a lot of properties everywhere, that could benefit from map-syntax, but should verify it first.#2020-05-3013:25ikitommiwill just push the helpers into malli.#2020-05-3013:43ikitommihttps://github.com/metosin/malli/pull/202#2020-05-3015:29ikitommimerged. Also, added a docs/tips.md to collect stuff from here. All tips, docs welcome! https://github.com/metosin/malli/blob/master/docs/tips.md#2020-05-3015:30ikitommiAsciidoc might be a better format as it has the table of contents etc.#2020-05-3103:34naomarikIt took a little while to figure this out by examining malli source, is this how you guys are passing in outside vars to the error/fn property?
(register! :phone [:re
{:description "Phone",
:phone-prefix-regex phone-prefix-regex
:phone-code-regex phone-code-regex
:error/fn
'(fn [{:keys [value schema]} _]
(let [{:keys [phone-prefix-regex
phone-code-regex]} (m/properties schema)
has-prefix? (re-find phone-prefix-regex value)
has-code? (re-find phone-code-regex value)]
(cond
(not has-prefix?)
:errors/phone-invalid-prefix
(not has-code?)
:errors/phone-invalid-country-code
:else
:errors/phone-invalid-number)))}
phone-regex])#2020-05-3103:39naomarikMaybe would be helpful in the tips section 🙂#2020-05-3104:03naomarikActually this doesn't work in clojurescript. (m/properties schema) returns #object[:cljs.coreMetaFn] and don't know what to do with that.#2020-05-3104:32naomarikI noticed (sci.core/eval-string "(inc 1)") was returning the same MetaFn object and not 2 like the sci docs says it should. Just tried using the latest sci dependency and the error/fn works as expected. @ikitommi would you expect any issues using the latest sci with the current implementation of malli? Will do more testing on my end, I'm using the exact same validations for frontend and backend.#2020-05-3105:56ikitommi@naomarik I would expect latest sci works ok.#2020-05-3106:04ikitommiAlso, if you define the validations as cljc code, e.g. no need to serialize over the wire, you can use plain functions instead of fn source code:
(defn valid? [x] (and (int? x) (> x 10)))
`(m/validate [:fn valid?] 11)
; => true`
#2020-06-0218:37naomarikHey @ikitommi,
I have all my malli specs in a cljc file, already.
If I pass in functions instead of quoted functions, the clojurescript side won't through SCI will it? If that's the case will sci dependency get removed with advanced builds?#2020-06-0307:16naomarik@ikitommi Sorry was being lazy about this didn't try it yet... Just tested myself and SCI adds about 0.5 megs to the (non gzipped) output and 93kb to the final JS output with gzip -9. I'm not sending code to be evalled as string in my project, is it cool if we make SCI an optional require?#2020-06-0307:18naomarikI can try to do that myself but not sure how clean my solution would be compared to what you can come up with. Let me know what you think. I think I could also help with the docs to state that if sending functions over the wire is being used then SCI will be included for a heavier JS file.#2020-06-0307:20naomarikI think this function
(defn ^:no-doc eval [?code]
(if (fn? ?code) ?code (sci/eval-string (str ?code) {:preset :termination-safe
:bindings {'m/properties properties
'm/name name
'm/children children
'm/map-entries map-entries}})))
if we conditionally require sci if the if test fails the fn? check would be good.#2020-06-0307:50naomarikJust tried, conditional requires seem to not work, not sure what I was expecting requiring a library at runtime :)#2020-05-3106:05ikitommithere could also be an option to add custom bindings to sci, so one could use own fn's as part of the sci source code.#2020-05-3106:09ikitommibasically just merge to existing bindings https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L878-L883#2020-05-3110:41naomarikYeah adding to the list of sci bindings would have been the most obvious thing to do for me . I saw it wasn't parameterized though so kept investigating. Actually I didn't try the error/fn property without quoting it, passing in a function is a lot better in my use case. All the examples show quoted functions in the github page, so didn't think to try that.#2020-05-3111:47ikitommiDoc updates welcome#2020-06-0213:31ikitommiGreat talk by @plexus about malli: https://youtu.be/ww9yR_rbgQs#2020-06-0213:31ikitommiand the event today: https://clojureverse.org/t/new-online-meetup-clojure-european-summer-time/6018/6#2020-06-0214:02katoxJust a thought: one thing that makes debugging Malli structures hard is that it doesn't retain any information of how it was composed. If you def a SubPart and then def a Part based on that using mu/union that info is lost. Not that it would always be possible (because schemas can just be created programatically) but it might be useful to stuff something into Meta to display it later in the explain.#2020-06-0415:20Vincent CantinIn Minimallist, there are :let and :ref nodes which make composition of models explicit.
Maybe Malli will get something similar once it supports recursion.#2020-06-0718:33ikitommiArne had a custom :ref too.#2020-06-0718:34ikitommithere is also some thinking in the README about entities & values: https://github.com/metosin/malli#entities-and-values#2020-06-0718:34ikitommiHi all. Goal is to get malli out of pre-alpha before my summer vacations = this month.#2020-06-0718:35ikitommistarted with adding some new Protocols: https://github.com/metosin/malli/pull/204#2020-06-0806:44ikitommifirst class :string coming up:
(m/validate :string "")
; => true
(m/validate [:string {:min 1}] "")
; => false
(json-schema/transform [:string {:min 1, :max 4}])
; => {:type "string", :minLength 1, :maxLenght 4}
(mg/sample [:string {:min 1, :max 4}])
; => ("RMmR" "5" "oL" "G7" "ENo" "cAh" "iOb" "jG" "l8" "kuo")
(-> [:map
[:a :string]
[:b [:string {:min 1}]]
[:c [:string {:max 4}]]
[:d [:string {:min 1, :max 4}]]]
(m/explain
{:a 123
:b ""
:c "invalid"
:d ""})
(me/humanize))
;{:a ["should be string"],
; :b ["should be at least 1 characters"],
; :c ["should be at most 4 characters"],
; :d ["should be between 1 and 4 characters"]}#2020-06-0807:08ikitommihttps://github.com/metosin/malli/pull/205#2020-06-0807:37naomarikPerfect. I accidentally had a string? predicate in my spec awhile ago allowing unbounded inputs.#2020-06-0809:41pithyless@U055NJ5CC any chance for a blank? option? Or is that too specific to add to malli-core? It's the first thing I have to add to all my string specs, since you almost never want to allow user input of just blank characters. In a similar vein, do you think malli-core generator should be using char-alphanumeric or simply string? I understand the first gives you nicer looking example data, but the second is more useful in testing edge cases.#2020-06-0810:06pithylessOn second thought, the latter question is probably not worth the hassle (and one can always write a custom gen where it's needed). But still curious about a possibility of blank? or present? option.#2020-06-0810:07pithylessI moved my question to the PR, where it is probably more appropriate.#2020-06-0906:43ikitommi@U05476190 chaned the generator to char & string.#2020-06-0818:50ikitommiExample string-trimmer here too:
(require '[malli.transform :as mt])
(require '[malli.core :as m])
(require '[clojure.string :as str])
(defn string-trimmer []
(mt/transformer
{:decoders
{:string
{:compile (fn [schema _]
(let [{:string/keys [trim]} (m/properties schema)]
(when trim #(cond-> % (string? %) str/trim))))}}}))
(m/decode [:string {:min 1}] " " string-trimmer)
; => " "
(m/decode [:string {:string/trim true, :min 1}] " " string-trimmer)
; => ""
(m/decoder :string string-trimmer)
; => #object[clojure.core$identity] ... no-op! :)#2020-06-0906:13ikitommiAs the renaming of m/name to m/type got only upvotes, going to change that: https://github.com/metosin/malli/pull/206. It’s a BREAKING CHANGE - both for the public api & for people who have extended the Schema protocol. Sorry for that.#2020-06-0907:09ikitommiMerged the protocol extensions (https://github.com/metosin/malli/pull/204) and realized that there is a breaking change, malli.generator/-generator is used by the new malli.generator/Generator protocol, so the old multimethod is called malli.generator/-schema-generator. Feedback welcome.#2020-06-0907:27ikitommistarted to track the before-release changed into https://github.com/metosin/malli/blob/master/CHANGELOG.md#unreleased, as some are breaking, for the better.#2020-06-1107:22pithylessI'm struggling to wrap my head around custom transformers and validators; is there some way to create a pipeline of validate -> transform -> validate (ala spec/conform ?) Or when defining a transform interceptor, can I in some way tell the the chain to stop executing further since some assumption is invalid?#2020-06-1110:41ikitommicurrently no way to stop the chain in interceptors @pithyless, could be added easily with reduced. There are Schemas that short-circuit in -transform like the :or.#2020-06-1110:42ikitommiin reitit-malli, there is a Corcion protocol and impl which does transform + validate for requests and transform + validate + transform for responses.#2020-06-1110:42ikitommifor responses, it’s basically doing:
1. apply defaults, strip extra keys
2. validate the result
3. encode to whatever (string, json, xml)#2020-06-1110:43ikitommiIf you have some minimal sample, could take a look.#2020-06-1111:23pithyless^ the reitit Coercer case is the kind of thing I have in mind. I've been going back and forth on this, but malli has been forcing me to rethink my approach: perhaps, I should just assume there is only one canonical form of data inside my project runtime, and only at the boundaries do all the encoding/decoding via custom protocols (i.e. a custom Coercer protocol and implementation for reitit, datomic, etc.)#2020-06-1111:33ikitommithe reitit code is here: https://github.com/metosin/reitit/blob/master/modules/reitit-malli/src/reitit/coercion/malli.cljc. Basically, at route(r) creation time, for each Schema, a reified Coercion is created, storing encoder, decoder, validator and explainer for that schema. At runtime, it’s just a calling those (optimized fns) in whatever order suits for the use case.#2020-06-1206:10ikitommi#2020-06-1206:10ikitommiinitial bundle-size anaysis for cljs. related to #203#2020-06-1209:36borkdude@ikitommi Note that about 200kb of the sci bundle is coming from docstrings 🙂#2020-06-1209:37borkdudeat least, if you bump to the newest sci#2020-06-1210:07ikitommiyes, updated the latest. Haven’t really studied how the dce works and how can libraries be smaller. Would be great if one could drop off whole protocol method impls if the protocol method was not used. E.g. if you don’t have any calls to the transformers, we coud remote the -transformer methods, which is easily 30% of the whole malli.core.#2020-06-1210:07ikitommimost of the concerns are in separate namespaces (json-schma, human errors, inferring), but the important ones are in the core right now. But not always important 🙂#2020-06-1210:08ikitommialso, would be great to have a sci-liteversion without the docstrings.#2020-06-1210:09borkdudemaybe that can be accomplished using a goog/define#2020-06-1210:10borkdude:closure-defines {"sci.foo/DOCSTRINGS" false}#2020-06-1210:11borkdudemade an issue for it: https://github.com/borkdude/sci/issues/352#2020-06-1407:17ikitommiGoing to change how the registry is composed. Goal is to allow easy bootstrapping of custom registries. Allows much smaller bundle size on cljs and one can (hopefully) easily add custom registries.#2020-06-1407:21ikitommifew sample sizes (gzipped), for malli.core (just validation):
• current (full): 12kb
• just :map , :vector and :string: 4kb
• just :string: 700b#2020-06-1407:25ikitommiI was surprised that the closure dce actually removes all the protocol method impls too if there is nothing using those methods. e.g. not using m/decode or m/encode => all -transformer impls are removed under :advanced. This is really, reaaly, great.#2020-06-1407:26ikitommiapp that uses all the features from malli, might still be big, e.g. validation, human errors, transformation, programming with schemas etc.#2020-06-1418:36naomariknice!#2020-06-1418:36naomarikone step closer to malli frontend domination#2020-06-1610:45ikitommi#2020-06-1610:46ikitommiwith default, malli is all-immutable, with a CLOSURE_DEFINES or JVM Prop setup, the default registry can be swapped out in the user space.#2020-06-1610:47ikitommiallows also really smaller bundle sizes, moved all the defs into defns so closure can eliminate those if needed.#2020-06-1610:48ikitommi@plexus, would that help with your acme-schema sample?#2020-06-1610:48ikitommialso, the options would be default (immutable) or managed (swappable). any suggestions for better names?#2020-06-1611:29plexusyeah I think that would help us#2020-06-1614:21ikitommi#2020-06-1614:22ikitommieven more evil. optional MUTABLE registry constructor, just pass in your own atom.#2020-06-1616:04Vincent Cantinwould it be possible to avoid having any mutable registry by having the user define his own API functions defaulting to their own registry?#2020-06-1616:05Vincent Cantinmaybe via a macro? (def-malli-api my-registry)#2020-06-1617:03ikitommithat is one approach, and can be done on top of the current api. Not a fan of macros generating functions personally. Need extra work to work with static analysis tools (like cursive) Also, the api should require all the malli namespaces to be complete.#2020-06-1617:12ikitommiI'll make set-default-regisyry! to throw if the registry swapping is not enabled.#2020-06-1804:17ikitommimerged the registries in.#2020-06-1804:18ikitommiquick poke on the recursive schemas, using :ref:
(def ConsCell
[:maybe {:id :cons}
[:tuple int? [:ref :cons]]])
(m/explain ConsCell [1 [2 [3 [4 nil]]]])
; => nil
(m/explain ConsCell [1 [2 [3 [4]]]])
;{:schema [:maybe {:id :cons} [:tuple int? [:ref :cons]]],
; :value [1 [2 [3 [4]]]],
; :errors (#Error{:path [2 2 0 2 2 0 2 2 0 2],
; :in [1 1 1],
; :schema [:tuple int? [:ref :cons]],
; :value [4],
; :type :malli.core/tuple-size})}#2020-06-1805:18viestiNice! :)#2020-06-1805:25Vincent Cantindoes it still work with cross recursion ?
ping refers to pong, and pong refers to ping#2020-06-1804:51ikitommiok, need to figure out how things like -accept should work with recursive schemas as it’s eager (validation and explain can be both lazy). I guess need to keep a local book keeping not to walk over already walked :refs so that it doesn’t go to infinite loop.#2020-06-1805:23Vincent Cantin@ikitommi welcome to the club of infinite loop avoiders :-)#2020-06-1805:45ikitommidoes not work yet with mutual recursion, but I think we need :registry schema anyway, like there is :definitions for JSON Schma. Needed anyway for persisting the things. maybe something like:
(explain
[:registry
{:schemas [[:maybe {:id :ping} [:tuple [:eq "ping"] [:ref :pong]]]
[:maybe {:id :pong} [:tuple [:eq "pong"] [:ref :pong]]]]}
[:ref :pong]]
["ping" ["pong" ["ping" nil]]])
; => true#2020-06-1805:47ikitommibtw, a colourful discussion at the old draft PR: https://github.com/metosin/malli/pull/117#2020-06-1816:07pithylessI've got
:closure-defines {malli.registry/type "custom"}
in my shadow-cljs.edn, but still seeing error:
can't set default registry {:type "default"}
Any ideas what I may be forgetting?#2020-06-1816:10pithylessoh, that's weird - some kind of caching bug; forcing a recompile (after it showed the error) and it went away. ¯\(ツ)/¯#2020-06-1816:17pithylessThe new registry setup allows for easier bootstrapping custom code. :thumbsup:#2020-06-1817:14jkentit doesn’t look like the new first class :string is bundled with the latest reitit [metosin/reitit-malli "0.5.2"] is there reason for this or am I missing something?#2020-06-1818:02pithylessmetosin/reitit-malli on clojars is dependent on malli 0.0.1-SNAPSHOT - theoretically, last version was uploaded yesterday, but maybe you have some caching issues? (I think clojure deps caches any snapshot locally for 24(?) hours). Perhaps try to update your deps to depend directly on latest malli from master?#2020-06-1908:04pithylessI think there may be some kind of race-condition bug with the malli.registry/type code in closure-defines. Once in a while (after a clean shadow restart) I see the {:type "default"} error, but it works if I "force compile" in shadow.#2020-06-1908:24ikitommi@pithyless that is weird, might be a bug in #shadow-cljs ?#2020-06-1908:27ikitommi@jkent I'll release a new reitit version next week, master is build against the latest malli.#2020-06-1918:25eskosDefault update policy for snapshots in pomegranate is daily; that is, once per 24 hours. If you're using Leiningen, see https://github.com/technomancy/leiningen/blob/stable/sample.project.clj#L115-L117 and if you're using something else, I have no idea 🙂#2020-06-1918:26eskosOh that was weird, got a whole screen of text after typing that... Slack's lagging. Meant this as continuation to @pithyless#2020-06-2019:23dcjA month or so ago I used Malli for a project, and with recent changes to Malli, my code broke:
(def predicate-registry
(-> m/predicate-registry
(-register-var #'zoned-date-time?)
(-register-var #'local-date?)))
(def registry
(merge predicate-registry m/class-registry m/comparator-registry m/base-registry))
So now the -registry is -schemas, but then my -register-var calls fail.
I'll keep working on this, but any ideas?#2020-06-2020:03dcjOK, this worked:
(def registry
(merge (m/predicate-schemas)
(m/class-schemas)
(m/comparator-schemas)
(m/base-schemas)
{:zoned-date-time (m/fn-schema :zoned-date-time #'zoned-date-time?)
:local-date (m/fn-schema :local-date #'local-date?)}))
And then I changed my schema references from zoned-date-time? to :zoned-date-time, etc, and also changed the associated transformer encoder/decoder names#2020-06-2020:30ikitommi@dcj we are doing final code polishing to get the non-pre-aloha out. The breaking changes are listed in https://github.com/metosin/malli/blob/master/CHANGELOG.md#unreleased. Also, if you are using custom schemas, you should read https://github.com/metosin/malli/blob/master/README.md#schema-registry. Lot of options, and supports properly dce on cljs now#2020-06-2207:02ikitommiAbout recursive schemas. It seems that because of those, should to to add a (-children [this]) method into Schema protocol. Why? Currently, there is generic children function that returns the Schema AST for the children. The AST is unaware of any instance bindings such as local recursion targets. Given a Schema:
[:maybe {:id :cons}
[:tuple int? [:ref :cons]]]
, the generator for :tuple just sees children of (int? [:ref :cons])and doesn’t know what :cons refers to and fails.#2020-06-2207:05ikitommiWe could hack around this by pulling out options that created :tuple and using those to re-create the child schemas, providing all the needed local context. But, this is error-prone as one needs to wire-up all generators, visitors etc. using the original options from the father schema.#2020-06-2207:05ikitommi(currently passing the local context via options, which seems to be a good way to do it)#2020-06-2207:06ikitommiAnyway, this works now:#2020-06-2207:06ikitommi#2020-06-2207:07ikitommiwith recursion, one can set the recursion limit, just like with spec. One setting, but each :ref is counted separatly, defaulting to 10.#2020-06-2207:58ikitommiwrote more to the recursion issue, comments welcome: https://github.com/metosin/malli/pull/117#issuecomment-647348039#2020-06-2214:06Vincent CantinI feel that having more than 1 way of defining recursive structures implies that users will have to choose which to use.
Choice can sometimes be a problem more than a solution.#2020-06-2305:45ikitommimutual recursion, current status:
(mg/generate
::ping
{:registry (mr/composite-registry
(m/default-schemas)
{::ping [:maybe [:tuple [:= "ping"] [:ref ::pong]]]
::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]})
:size 7, :seed 86})
; => ["ping" ["pong" ["ping" ["pong" ["ping" nil]]]]]
tested also a fail-fast on ambiguity with refs, so instead of:
(mg/generate
::ping
{:registry (mr/composite-registry
(m/default-schemas)
{::ping [:maybe {:id ::pong} [:tuple [:= "ping"] [:ref ::pong]]]
::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]})
:size 7, :seed 86})
; => ["ping" ["ping" ["ping" ["ping" ["ping" nil]]]]]
the default code will throw instead:
Execution error (ExceptionInfo) at malli.core/fail! (core.cljc:80).
:malli.core/ambiguous-ref {:type :ref, :ref :user/pong}#2020-06-2305:46ikitommithe code is in a new PR: https://github.com/metosin/malli/pull/209#2020-06-2306:48Vincent Cantin@ikitommi [suggestion] if you can embed registries locally, then maybe you don't need to have support for a custom global registry - just ask the users to include their registry as a part of their models.#2020-06-2306:59ikitommiThe custom Schema elements can’t be embedded as data, if they are code. It’s good have a mechanism to add those to the schema registry. All user/project-defined that are just data COULD be used via embedded registries, a decision done in user space.#2020-06-2307:02ikitommiI woudn’t register any project-spesific (data) schemas into global registry, as they can be references mosty as Vars.#2020-06-2307:05ikitommiThe recursive schemas could also be allowed to be introduces using Vars, but not sure if that’s a good idea.#2020-06-2307:06ikitommiSomething like:
(declare Pong)
(def Ping [:maybe [:tuple [:= "ping"] [:ref #'Pong]]])
(def Pong [:maybe [:tuple [:= "pong"] [:ref #'Ping]]])
#2020-06-2307:09ikitommiImaginary example with not much boilerplate with schematized fns using custom schema element :db/ref registered into the global registry:
(m/defn my-fn [x :- int?, y :- [:maybe [:db/ref uuid?]]
(println x y))
#2020-06-2307:14ikitommiAbout the Var refs - how would that be serialized? It could be done by having an asymmetric m/form for that component, which might be a bad idea:
(m/form [:maybe [:tuple [:= "ping"] [:ref #'Pong]])
;[:registry
; {:registry {::ping [:maybe {:id ::pong} [:tuple [:= "ping"] [:ref ::pong]]]
; ::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]}}
; [:ref ::ping]]#2020-06-2307:14ikitommior is it?#2020-06-2307:47ikitommithis works now too:
(mg/generate
[:registry
{:registry {::ping [:maybe [:tuple [:= "ping"] [:ref ::pong]]]
::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]}}
[:ref ::ping]]
{:size 7, :seed 86})
; => ["ping" ["pong" ["ping" ["pong" ["ping" nil]]]]]#2020-06-2507:28ikitommilocal registries, holes, nesting and masking - the recent thoughts from the hammock, comments most welcome: https://github.com/metosin/malli/pull/117#issuecomment-649305848#2020-06-2811:23ikitommiAbout to add eager references too:
(def Schema
[:and
{:registry {::a ::b
::b ::c
::c [:schema pos-int?]}}
[:and ::a ::b ::c]])
(m/form Schema)
;[:and {:registry #:user{:a :user/b
; :b :user/c
; :c [:schema pos-int?]}}
; [:and :user/a :user/b :user/c]]
(m/to-map-syntax Schema)
;{:type :and
; :properties {:registry #:user{:a :user/b
; :b :user/c
; :c [:schema pos-int?]}}
; :children [{:type :and
; :children [{:type :schema
; :children [:user/a]}
; {:type :schema
; :children [:user/b]}
; {:type :schema
; :children [:user/c]}]}]}#2020-06-2811:25ikitommie.g. each registry hop retains the information the original linkage + :schema element to mark Entitys in a schema data graph.#2020-06-2811:25ikitommi(all registry hops are internaly :schema too)#2020-06-2811:26ikitommiaccept can be configured to automatically walk over those, if wanted.#2020-06-2811:26ikitommiboth :schema and :ref also implement the new
(defprotocol RefSchema
(-deref [this] "returns the referenced schema"))
#2020-06-2812:16ikitommi… and for the given schema, the validation fn is just pos-int?#2020-06-2812:16ikitommi(m/validator Schema)
; #object[clojure.core$pos_int]
#2020-06-2812:16ikitommiwhich is kinda awesome 🙂#2020-06-3010:00borkdude@ikitommi https://github.com/metosin/malli/pull/210#2020-06-3010:03borkdudeI made a PR to malli which makes sci optional.
The tests are failing, but this is only because the order in which namespaces are loaded is random. sci.core needs to be loaded first. But since sci.core is now optional, the tests have to be refactored accordingly anyway. Just putting this here in case someone has ideas about it.#2020-06-3010:03borkdudeLocally the tests pass, FWIW.#2020-06-3010:19borkdudeI might put that dynaload code into a library so it can be used in more projects.#2020-06-3012:49ikitommiwill check that out, thanks!#2020-06-3012:49ikitommia separat one-purpose lib would be good#2020-06-3012:49ikitommimerged recursion PR into master.#2020-06-3012:50ikitommihere’s a sample (the generation is blocking the main thread, should port to use web-worker?): https://malli.io/?value=%7B%3Alines%20%5B%7B%3Aburger%20%7B%3Aname%20%22NAUGHTY%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Adescription%20%22Finnish%20100%25%20beef%20patty%2C%20cheddar%2C%20St%20Agur%20blue%20cheese%2C%20bacon%20jam%2C%20rocket%2C%20aioli%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Aorigin%20%7B%3Aname%20%3AFI%7D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Aprice%2011%7D%2C%0A%20%20%20%20%20%20%20%20%20%20%3Aamount%202%7D%5D%2C%0A%20%3Adelivery%20%7B%3Adelivered%20false%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%3Aaddress%20%7B%3Astreet%20%22H%C3%A4meenkatu%2010%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Azip%2033100%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Acountry%20%7B%3Aname%20%3AFI%2C%20%3Aneighbors%20%5B%7B%3Aname%20%3APO%7D%5D%7D%7D%7D%7D&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%3Auser%2Fcountry%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20%5B%3Aenum%20%3AFI%20%3APO%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aneighbors%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aoptional%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Avector%20%5B%3Aref%20%3Auser%2Fcountry%5D%5D%5D%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Fburger%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20string%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Adescription%20%7B%3Aoptional%20true%7D%20string%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorigin%20%5B%3Amaybe%20%3Auser%2Fcountry%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprice%20pos-int%3F%5D%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Forder-line%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aburger%20%3Auser%2Fburger%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aamount%20int%3F%5D%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Forder%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Alines%20%5B%3Avector%20%3Auser%2Forder-line%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Adelivery%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Adelivered%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aaddress%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Astreet%20string%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Azip%20int%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acountry%20%3Auser%2Fcountry%5D%5D%5D%5D%5D%5D%7D%7D%0A%20%3Auser%2Forder%5D#2020-06-3015:28borkdude@ikitommi https://github.com/borkdude/dynaload#2020-06-3016:43SathiyaHi. I have an use case with the data structure and scenario similar to the following example
{
"name": string,
"age": int,
"married": boolean,
"sex": string
"partner": {
"husband":{"name": string, "age": int},
"wife":{"name": string, "age": int}
}
}
name in husband is mandatory if the status is married and sex is Male. name in wife is mandatory if status is married and sex is female. both are not needed if married is false. how do i write the malli schema validations for the following scenario. if both the fields are at the same level i could add a function and validation. since one is at a parent and the other field is inside the child, i am unable to find a solution to implement this. any help would be appreciated. thanks in advance#2020-06-3017:29ikitommihi @kspriyan31. You can add rules to top-level and add error metadata to point into nested fields. Something like:
(def Person
[:and {:registry {:partner [:map ["name" string?] ["age" int?]]}}
[:map
["name" string?]
["age" int?]
["married" boolean?]
["sex" string?]
["partner" {:optional true}
[:map
["husband" {:optional true} :partner]
["wife" {:optional true} :partner]]]]
[:fn {:error/message "wife name is mandatory for married men"
:error/path ["partner" "wife" "name"]}
(fn [{:strs [sex married partner]}]
(not (and married (= "male" sex) (not (get-in partner ["wife" "name"])))))]
[:fn {:error/message "husband name is mandatory for married female"
:error/path ["partner" "wife" "name"]}
(fn [{:strs [sex married partner]}]
(not (and married (= "female" sex) (not (get-in partner ["husband" "name"])))))]])
(-> Person
(m/explain {"name" "Mauno"
"age" 31
"married" true
"sex" "male"
"partner" {"wife" {"age" 41}}})
(me/humanize))
;{"partner" {"wife" {"name" ["missing required key"
; "wife name is mandatory for married men"]}}} #2020-06-3017:34ikitommihere’s the same in http://malli.io: http://shorturl.at/hmuFV#2020-06-3017:38SathiyaThank you @ikitommi#2020-06-3017:39ikitommiif you change the sex to [:enum "male" "female" "other"]the sample value generation can generate better values:
({"name" "",
"age" 1,
"married" true,
"sex" "male",
"partner" {"husband" {"name" "", "age" -1}, "wife" {"name" "", "age" 0}}}
{"name" "", "age" 2, "married" false, "sex" "male"}
{"name" "K5", "age" 2, "married" false, "sex" "male", "partner" {"wife" {"name" "", "age" -1}}}
{"name" "", "age" 3, "married" false, "sex" "male"}
{"name" "8Mn",
"age" 3,
"married" true,
"sex" "male",
"partner" {"husband" {"name" "J", "age" -4}, "wife" {"name" "", "age" 3}}}
{"name" "bsyt4", "age" 1, "married" false, "sex" "female"}
{"name" "8k5yk", "age" 6, "married" false, "sex" "female", "partner" {"husband" {"name" "4", "age" 13}}}
{"name" "5ut0D2sm",
"age" 1,
"married" true,
"sex" "female",
"partner" {"husband" {"name" "", "age" -102}, "wife" {"name" "i4Q5l6xa", "age" 0}}}
{"name" "", "age" 1, "married" false, "sex" "male"}
{"name" "l", "age" 3, "married" false, "sex" "male", "partner" {"husband" {"name" "Ixh4drJNb", "age" -37}}})#2020-06-3017:39ikitommiyour welcome#2020-07-0120:47katoxI'm looing at https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj
How can I enable the same kind of coercion but to path-params instead of query params? If I change just the key it goes through untouched.#2020-07-0123:34jkent@katox here’s an example of path coercion
["/foo/:x/:y"
{:get {:summary "plus with malli query parameters"
:parameters {:path [:map [:x int?] [:y int?]]}
:handler (fn [{{{:keys [x y]} :path} :parameters}]
{:status 200
:body {:total (+ x y)}})}}]#2020-07-0208:02katox@jkent of course, a shorter key than variants I tried. Thank you, it works!#2020-07-0210:50ikitommiAlmost there:
(def Order
[:schema
{:registry {"Country" [:map
[:name [:enum :FI :PO]]
[:neighbors [:vector [:ref "Country"]]]]
"Burger" [:map
[:name string?]
[:description {:optional true} string?]
[:origin [:maybe "Country"]]
[:price pos-int?]]
"OrderLine" [:map
[:burger "Burger"]
[:amount int?]]
"Order" [:map
[:lines [:vector "OrderLine"]]
[:delivery [:map
[:delivered boolean?]
[:address [:map
[:street string?]
[:zip int?]
[:country "Country"]]]]]]}}
"Order"])
(malli.mermaid/class-diagram Order)
; classDiagram
; class Country {
; :name [:enum :FI :PO]
; :neighbors [:vector #:gen{:max 0} [:ref Country]]
; }
; class Burger {
; :name string?
; :description string?
; :origin [:maybe Country]
; :price pos-int?
; }
; class OrderLine {
; :burger Burger
; :amount int?
; }
; class Order {
; :lines [:vector OrderLine]
; :delivery Order_Delivery
; }
; class Order_Delivery_Address {
; :street string?
; :zip int?
; :country Country
; }
; class Order_Delivery {
; :delivered boolean?
; :address Order_Delivery_Address
; }
; Country o-- Country
; Burger o-- Country
; OrderLine o-- Burger
; Order o-- OrderLine
; Order *-- Order_Delivery
; Order_Delivery_Address o-- Country
; Order_Delivery *-- Order_Delivery_Address#2020-07-0210:50ikitommi#2020-07-0210:53ikitommias the map keys can be anything, the registry keys can be any non-vector. Strings looks nice.#2020-07-0210:58ikitommiwhen a Schema is created (using m/schema), all the registry refs are eagerly validated and the current values are captured:
• have an Invalid ref? -> fail early
• using mutable registry? -> the created schema instance (including refs!) is still immutable
• wan’t to know the local registry which the schema was created with -> accessible via m/options#2020-07-0211:00ikitommiclojure.spec with mutable global registry, fail late and mutable refs doesn’t seem that right anymore.#2020-07-0219:50kwrooijenIs this intended behavior?
Here we say that the :id field is optional in the map schema. Which works
(m/validate
[:map
[:id {:optional true} int?]]
{}) ; => true
Here we give the :id field a default value. But this doesn't work
(m/decode
[:map
[:id {:default 42} int?]]
{}
mt/default-value-transformer) ; => {}
Instead, we have to give field's spec a default, not the map's key
(m/decode
[:map
[:id [:and {:default 42} int?]]]
{}
mt/default-value-transformer) ; => {:id 42}
Personally I find the last example a bit strange (Or better yet, I find it strange that the second doesn't work). I'm not saying the ints have a default, I'm saying that this maps :id field has a default. Just like how :id is an optional field, not that that int? is an optional schema.
Maybe there's a reason it's designed this way? I could just be looking at this from the wrong angle.#2020-07-0219:52kwrooijenI guess it's more that I find it very unexpected that the second example doesn't work#2020-07-0307:27ikitommi@kevin.van.rooijen handling map-entry properties is the last issue to be resolved to make the malli design complete for release, see https://github.com/metosin/malli/issues/116#2020-07-0307:29ikitommiHave some ideas how to fix that elegantly, but comments on the issues most welcome#2020-07-0307:29kwrooijenAh ok, good to hear#2020-07-0307:29kwrooijenI did some digging and realized I asked the same question 3 months ago haha (but no answer at that time)#2020-07-0307:30ikitommiHmm.. could you link that with #86?#2020-07-0307:31ikitommior do you mean here? Slack history kinda sucks#2020-07-0307:31kwrooijenApparently I asked that question in my PR to support qualified keywords in maps#2020-07-0307:32kwrooijenI had a bit of a monologue in that pr haha#2020-07-0307:33kwrooijenhttps://github.com/metosin/malli/pull/194#issuecomment-609977835#2020-07-0307:34ikitommithere is a bunch of PRs I know are related to this, wanted to have this figured out before merging any of those in. Sorry for the lag with PRs.#2020-07-0307:35kwrooijenDon't worry about it, it's a tough problem to handle#2020-07-0311:32sudakatuxHI.#2020-07-0311:32sudakatuxSo what would be the correct type for a date LocalDateTime to be more specific#2020-07-0311:32sudakatux:responses {200 {:body [:vector
[:map
[:id int?]
[:sender_id int?]
[:conversation_id int?]
[:created_at string?]
[:state string?]]
]} #2020-07-0311:33sudakatuxI tried inst? and it did not work. the error shows it as a string because of the conversion. but the validation seems to run before#2020-07-0311:38sudakatux{:id 1,
:message "string",
:sender_id 1,
:conversation_id 1,
:created_at #object[java.time.LocalDateTime 0x4cc366e7 "2020-06-30T18:50:23.695073"],
:state "UNREAD"}#2020-07-0312:02ikitommihttps://github.com/metosin/malli/issues/49 is the issue for adding proper (Java8+ & goog.*) dates.#2020-07-0312:03ikitommibefore that, [:fn (partial instance? LocalDateTime)] does the validation part#2020-07-0312:24sudakatuxcool thanks#2020-07-0417:29ikitommistarted on thinking https://github.com/metosin/malli/pull/212#2020-07-0608:15steveb8nQ: I’m starting with Malli in my re-frame client/forms. just checking to see if there’s any gotchas/advice from the team here?#2020-07-0608:51Vincent CantinIt depends how you use Malli .. validation and coercion?#2020-07-0609:24steveb8njust validation for now. keeping it simple to start. schemas hardcoded in client code. later I intend to send schemas over the wire from the server#2020-07-0609:25steveb8nI’m using it as a validator for https://github.com/luciodale/fork in this iteration. thus no coercion since the form inputs will constrain the values#2020-07-0609:31ikitommionly notice is the bundle size, few hours away from making sci optional. Before that, the bundle is 200+kb. After, it's 1-12kb gzipped.#2020-07-0609:32ikitommioh, and you should create validators ahead of time, running m/validate can be orders of magnitude slower than just calling the returned function from m/validator.#2020-07-0609:33ikitommisame applies to m/explain vs m/explainer and to m/decode & m/decoder (and encoders, generators etc)#2020-07-0609:37ikitommifork-malli-example would be great gist/blog. Tried the first version of it, but was stumbling with nested data and for that case, did my own formik-kinda-thing for that project.#2020-07-0609:38ikitommilooking forward to seeing how the integration would look today @steveb8n !#2020-07-0609:39steveb8nthanks! great to know about the bundle increase.#2020-07-0609:39steveb8nand I’ll def the schemas for now since they are static#2020-07-0609:44steveb8nI’ll try and squeeze in a sample repo this week 🙂#2020-07-0610:13kwrooijenHey, is it possible to generate only alpha-numeric values with the :string type? Currently it will go wild and generate whatever it wants (which makes sense)#2020-07-0610:14kwrooijenOr is my only option to use a regular expression?#2020-07-0610:20ikitommi@kevin.van.rooijen you can use :gen/gen to override the generator, there is one for alphanumeric. Or use :gen/elements and list all valid in there.#2020-07-0610:21ikitommiI think :string could have format option too, like in JSON Schema. But doesn't have that yet. One format could be :alphanumeric , validator and generator would follow that...#2020-07-0610:25kwrooijenAh cool, the :gen/gen is what I need. And it would be nice to allow something like a :format option. Maybe make it extendable in some sort of way, perhaps a multimethod?#2020-07-0610:26kwrooijene.g.
[:string {:format :alphanumeric}]
(defmethod malli/format :alphanumeric [_ v]
,,,)
Something like this, or possibly more inline with how the registry for types works#2020-07-0701:07steveb8n@ikitommi sample repo up and running: https://github.com/stevebuik/fork-malli-ideas#2020-07-0701:07steveb8nfirst question to come from it: can the fast validation i.e. m/validator be used with m/explain?#2020-07-0704:50steveb8nand what was the “nested data” problem you had with this integration? maybe it’s another good demo in this repo#2020-07-0713:57jkentdoes malli support default error messages for enums? I’m getting "message": "unknown error" unless I do something like this for each enum:
(def source-system
[:enum
{:error/message "should be either: foo|bar"
:swagger/type "string"
:description "source system from where the request comes in"}
"foo" "bar"])#2020-07-0720:37ikitommi@jkent the default errors are defined in https://github.com/metosin/malli/blob/master/src/malli/error.cljc. There is no default for :enum, but could be. PR welcome!#2020-07-0720:38ikitommialso, if some more native english-writing person could check the spelling of the default errors, should should be int be should be an int etc…#2020-07-0802:52steveb8nhere you go….#2020-07-0802:52steveb8nhttps://github.com/metosin/malli/pull/215#2020-07-0802:53steveb8noops. broke the tests. fix coming#2020-07-0803:06steveb8nfixed!#2020-07-0721:03ikitommi@steveb8n just trying to update a value in path like [:user :address :street] , I recall it expected string paths and didn't support nesting. Might have been just user error.#2020-07-0801:25jkent@ikitommi great suggestion. here’s a PR to add a default error message for enums: https://github.com/metosin/malli/pull/214/commits#2020-07-0806:18ikitommiMerged both PRs, thanks!#2020-07-0907:36ikitommire-implemented the schema visitor using walker: m/accept is now a postwalk and and there is new m/find-first to do lazy scanning for schemas. Though of doing a prewalk too, but coudn’t find any valid use case for such and would be more complex for schema extenders to add support for it. Not sure how useful the m/find-first is and should merge this: https://github.com/metosin/malli/pull/216#2020-07-0907:39ikitommishould the m/accept be just m/walk in the future? or m/post-walk?#2020-07-0909:05ikitommi(deftest find-first-test
(let [schema [:map
[:x int?]
[:y [:vector [:tuple
[:maybe int?]
[:or [:and {:salaisuus "turvassa"} boolean?] int?]
[:schema {:salaisuus "vaarassa"} false?]]]]
[:z [:string {:salaisuus "piilossa"}]]]]
(let [walked-properties (atom [])]
(is (= "turvassa" (m/find-first
schema
(fn [s _in _options]
(some->> s m/properties (swap! walked-properties conj))
(some-> s m/properties :salaisuus)))))
(is (= [{:salaisuus "turvassa"}] @walked-properties)))
(let [walked-properties (atom [])]
(is (= "vaarassa" (m/find-first
schema
(fn [s _in _options]
(some->> s m/properties (swap! walked-properties conj))
(some-> s m/properties :salaisuus #{"vaarassa"})))))
(is (= [{:salaisuus "turvassa"}
{:salaisuus "vaarassa"}] @walked-properties)))))#2020-07-0909:45alpoxHi all! Is there a possibility to merge definitions declaratively - for example in an edn definition given that the definitions to merge are defined in a registry?#2020-07-0909:48alpoxI see I can do
[:and
[:ref :test/object]
[:ref :test/annotated]
[:map ....]]
but sadly this would test for all to be true rather than that later definitions override definitions from the former. Basically, I'm asking if there is something like mu/merge in a declarative way#2020-07-0912:23ikitommi@alpox nothing just now, but sounds like a good idea. Could you write an issue of that?#2020-07-0913:39alpox@ikitommi Done: https://github.com/metosin/malli/issues/217 thanks for the response!#2020-07-0914:35euccastrowhat's the idiomatic way to specify a nonempty string in a schema? should I actually use a custom registry as in the README?#2020-07-0914:39euccastrocurrently I'm doing this
(def nonempty-string
[:and
string?
[:fn {:error/message "should be nonempty string"}
(fn [x] (pos? (count x)))]])#2020-07-0914:42euccastroone minor problem with that is that malli/explain gives me two error messages: "should be string" and "should be nonempty string". I tried to set the :error/message in the :and instead to get only one error, but that didn't work#2020-07-0914:44euccastro(I'm using the malli version that reitit 0.5.2 requires, i.e. 0.0.1-20200525.162645-15, if that matters)#2020-07-0914:45euccastroI'm happy to upgrade to any version that is compatible with reitit 0.5.2#2020-07-0914:58ikitommi@U65FN6WL9 [:string {:min 1}], will need to release new version to work with reitit#2020-07-0915:15euccastrothank you!#2020-07-0915:23jkent@ikitommi is there any chance you can release a new version of reitit with the latest malli?#2020-07-0917:24euccastroanother option for early adopters would be to release cutting-edge versions of malli under a separate namespace and artifact id (e.g., malli.pre-alpha) so we can use that for our own code without fear of breaking reitit. that would also reduce the pressure to update reitit unless new malli functionality is actually needed there#2020-07-0917:24euccastroplease ignore this if it sounds like too much of a hassle 🙂#2020-07-1006:22ikitommiMorning. Pushed out new immutable & compatible versions into clojars, ping @euccastro @jkent
[metosin/malli "0.0.1-20200709.163702-18"]
[metosin/reitit "0.5.3"]
#2020-07-1007:57ikitommimerged the Walker PR into master, here’s the BREAKING:
* 10.7.2020
* `[metosin/malli "0.0.1-20200710.075225-19"]`
* **BREAKING:**: Visitor is implemented using a Walker.
* `m/accept` -> `m/walk`
* `m/schema-visitor` -> `m/schema-walker`
* `m/map-syntax-visitor` -> `m/map-syntax-walker`#2020-07-1207:58ikitommiAbout to replace MermaidJS with DOT, because it just works: https://github.com/metosin/malli/pull/219#2020-07-1207:59ikitommithe whole transformer is 69 loc, with most code reusable and could be part of malli.util (collecting and resolving references).#2020-07-1208:00ikitommithe actual DOT-transformer is ~30 loc 🙂#2020-07-1209:32shmuel buchnikHi I am new to malli
Can I use multi with recursive ?
I tried this and it does not work
(def map-multi-recursive
[:map
{:registry
{::filter
[:multi
{:dispatch :type}
["in" [:map [:type [:= "in"]] [:value [:vector [:ref ::filter]]]]]
["or" [:map [:type [:= "or"]] [:value [:vector [:ref ::filter]]]]]
["not" [:map [:type [:= "not"]] [:value [:ref ::filter]]]]
["in" [:map [:type [:= "in"]] [:value [:map
[:dimensions dimension]
[:values [:vector string?]]]]]]]}}
:filter ::filter])
#2020-07-1210:06ikitommi@shmuel.buchnic I believe the map entry should be [:filter ::filter], e.g. surround with brackets#2020-07-1210:06ikitommirecursion is a generic solution, unless there are bugs, it works with all Schemas#2020-07-1211:01shmuel buchnik@U055NJ5CC thanks for the fast response I tried and it still return an invalid schema .
I will try to narrow down the issue .#2020-07-1212:59shmuel buchnik@U055NJ5CC well it is working with simple validate , but trying to use it as
:coercion to body parameters and it fails .#2020-07-1213:00ikitommiyou seem to have two "in" branches in multi.#2020-07-1213:01ikitommiwill fail-fast in master: https://github.com/metosin/malli/commit/5d54e6b285c3ce440f1310302bdc2ba20a84d508#2020-07-1213:01ikitommi(m/schema
[:map
{:registry
{::filter
[:multi
{:dispatch :type}
["in" [:map [:type [:= "in"]] [:value [:vector [:ref ::filter]]]]]
["or" [:map [:type [:= "or"]] [:value [:vector [:ref ::filter]]]]]
["not" [:map [:type [:= "not"]] [:value [:ref ::filter]]]]
["in" [:map [:type [:= "in"]] [:value [:map
[:dimensions [:tuple int? int?]]
[:values [:vector string?]]]]]]]}}
[:filter ::filter]])
; => Throws :malli.core/non-distinct-entry-keys {:keys ("in" "or" "not" "in")}#2020-07-1213:02shmuel buchnikI fixed that the updated looks like this :
(def filter-local-registry
{:registry
{::filter
[:multi
{:dispatch :type}
["and" [:map [:type [:= "and"]] [:value [:vector [:ref ::filter]]]]]
["or" [:map [:type [:= "or"]] [:value [:vector [:ref ::filter]]]]]
["not" [:map [:type [:= "not"]] [:value [:ref ::filter]]]]
["in" [:map [:type [:= "in"]] [:value [:map
[:dimension dimension]
[:values [:vector string?]]]]]]]}})
#2020-07-1213:02ikitommiworks now?#2020-07-1213:03shmuel buchnikno 😞 I fixed it before u wrote forgot to update.
validate call is working#2020-07-1213:03shmuel buchnikbut as :coercion it does not work#2020-07-1214:05ikitommiwhat version are you using of reitit?#2020-07-1214:07shmuel buchnik0.5.3 I debug it .
It looks like at first he find filter in registry (holds 2 the default one and the local)
But on later call I only see the default on the registry and than lookup fail .#2020-07-1216:18shmuel buchnikNarrowing issue :
This is the schema
(def sample-request
[:map {:registry {::age [:and int? [:> 18]]}}
[:age ::age]])
This is the route
(def overview
["/api/v1/overview"
{:swagger {:tags ["Overview"]}
:post {:summary "get an overview data"
:parameters {:body sample-request}
:handler get-overview-data
}}])
The handler and other are just like in :
https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj
So it is not related to recursive or mutli I guess it is just wrong usage of me with registry I just did not find out what it is 😞#2020-07-1218:22ikitommiI can reproduce this.#2020-07-1218:23ikitommiWill check this out#2020-07-1308:10ikitommifixed in https://github.com/metosin/malli/commit/9f105a53e459f9b85bfbd5c07d747fbe5d5d6f65#2020-07-1308:11ikitommipushed out [metosin/reitit "0.5.4"] with the fix in.#2020-07-1306:23steveb8nI enhanced the Malli/Fork integration demo to use idiomatic keyword maps in the re-frame app-db and string keyword for Fork forms. @lucio#2020-07-1306:23steveb8nhttps://stevebuik.github.io/fork-malli-ideas/#!/basics#2020-07-1306:23steveb8nMalli transforms makes this easy to do#2020-07-1306:24steveb8nIt has raised another question. @ikitommi when you have a chance, can you comment on the 2 questions?#2020-07-1308:17ikitommi@steveb8n sure:
1. m/explainer returns a optimized pure function for explaining, e.g:
(m/explainer [:map [:x int?]])
2. mt/transformer can be used to compose transformers, runs them in single sweep. Also, you could store the computed m/decoder :
(m/decoder
[:map [:x int?]]
(mt/transformer (mt/key-transformer {:decode keyword}) (mt/string-transformer)))
#2020-07-1308:19steveb8nthanks. for #2 I tried the composition via mt/transformer but couldn’t get it to transform properly. I am adding complexity by using it with a “multi-schema”#2020-07-1308:19ikitommiat best, the m/decoder (or m/encoder) know there is nothing to do and return identity:
(m/decoder
[:map [:x int?]]
(mt/transformer
(mt/json-transformer)
(mt/json-transformer)
(mt/json-transformer)
(mt/json-transformer)))
; => #object[clojure.core$identity]#2020-07-1308:20steveb8nnot in the public repo, in my private app#2020-07-1308:20ikitommiIf you have and example of multi-schema that doesn’t work, please share, I could take a look#2020-07-1308:20steveb8nthanks. will do.#2020-07-1308:20steveb8nand I’ll try m/explainer as well 🙂#2020-07-1308:22ikitommitransformers are computed into interceptor chains, where the order matters.#2020-07-1308:23steveb8nI think that’s the right direction to solve it: ordering#2020-07-1308:23steveb8nbut I’ll start with the simpler (public) example#2020-07-1308:23ikitommialso, as the transformers are mounted from root schema towards leafs, the :multi decoding is applied before it’s childs (as we don’t know which child is selected), so you might need to decode the :multi dispatch key manually.#2020-07-1308:24ikitommithere is an example in the README:
(m/decode
[:multi {:dispatch :type
:decode/string '#(update % :type keyword)}
[:sized [:map [:type [:= :sized] [:size int?]]]
[:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
{:type "human"
:name "Tiina"
:age "98"
:address {:country "finland"
:street "this is an extra key"}}
(mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
;{:type :human
; :name "Tiina"
; :address {:country :finland}}#2020-07-1308:24steveb8nyep: tried that too. failed. I’ll dig deeper#2020-07-1308:25steveb8nI’ll get back to you. making this work makes the fork/malli integration nice and idiomatic on both sides#2020-07-1308:25steveb8nso worth the effort from me#2020-07-1308:27ikitommioh, noticed you have the custom error message for :enum: {:error/message "Must be London or Tampere"}. Enums have something decend nowadays by default: https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L63-L66#2020-07-1308:27ikitommimaybe that could be polished into having , & or s by default?#2020-07-1308:28steveb8nok. I’ll try that too. and I’ll send a PR for the english grammar if required#2020-07-1308:28steveb8nend of the day here (Sydney) so probably not till tomorrow#2020-07-1308:51ikitommiMy guess is that you should have a new named transformer, something like:
(mt/transformer
{:name :before}
(mt/key-transformer {:decode keyword})
(mt/string-transformer))
and in the :multi:
[:multi {:dispatch :type
:decode/before '#(update % :type keyword)} ...#2020-07-1308:53ikitommithis way, the :multi selects the branch correctly before anything else, and the key-transformer & string-transformer both work correctly.#2020-07-1323:18steveb8nok. here’s a simple test case from the demo repo….#2020-07-1323:19steveb8n(let [v {"id" "123"
"github-followers" "10"}
schema [:map
[:id :string]
[:github-followers pos-int?]]]
(->> (mt/transformer (mt/key-transformer {:decode keyword})
mt/string-transformer)
(m/decode schema v)
cljs.pprint/pprint))
why does the string transformer not work?#2020-07-1408:36ikitommi@steveb8n doesn’t work :thinking_face:. Will check it out.#2020-07-1408:37ikitommi(for some reason, the keys are not followed AFTER converted to keywords.#2020-07-1408:37ikitommioh, the key-transformer applies on :leave, which is after-the-fact.#2020-07-1408:39ikitommi… should be on :enter on decode and on :leave on encode. Just a sec.#2020-07-1408:58steveb8nno hurry. the workaround of doing 2 transforms is ok. but it will be good to see the transform composition handling this use case eventually#2020-07-1408:59steveb8neven if that means dropping down to the interceptor level#2020-07-1409:00steveb8nbtw: for fun I might run https://github.com/stevebuik/Stu on my project. it’ll show the extra 200k from sci very clearly#2020-07-1409:14ikitommi@steveb8n fixed in master: https://github.com/metosin/malli/commit/679ca7780f2e36e427885e2343772369e048598e#2020-07-1409:15steveb8ngreat! that was quick. I’ll update first thing tomorrow#2020-07-1409:16steveb8n@lucio just added a feature to fork at about the same speed. you guys are awesome#2020-07-1409:16ikitommididn’t know about Stu, looks nice#2020-07-1409:17steveb8nit needs a couple of fixes to use properly. let me know if you want to use it. then I’ll go fix them#2020-07-1409:46ikitommi@shortlyportly related to you question on #reitit, would this help:
(let [path->schema (atom {})]
(mu/find-first
[:and
[:fn '(constantly true)]
[:map
[:name string?]
[:tags [:set [:map [:name string?]]]]
[:address [:and
[:fn '(constantly true)]
[:map [:street string?]]
[:fn '(constantly true)]]]]
[:fn '(constantly true)]]
(fn [s i _]
(swap! path->schema update i (fnil identity s))
nil))
@path->schema)
;{[] [:and
; [:fn (constantly true)]
; [:map
; [:name string?]
; [:tags [:set [:map [:name string?]]]]
; [:address [:and [:fn (constantly true)] [:map [:street string?]] [:fn (constantly true)]]]]
; [:fn (constantly true)]],
; [:name] string?,
; [:tags] [:set [:map [:name string?]]],
; [:tags :malli.core/in] [:map [:name string?]],
; [:tags :malli.core/in :name] string?,
; [:address] [:and [:fn (constantly true)] [:map [:street string?]] [:fn (constantly true)]],
; [:address :street] string?}#2020-07-1409:47ikitommiyou would get the schemas per value path for standalone validation.#2020-07-1416:27Dave SimmonsThanks again @ikitommi - I'll take a look. Just wanted to say love your talks and the libraries metosin are putting out. many thanks.#2020-07-1507:54steveb8n@lucio @ikitommi I updated the demo app with the new composable transformers fix https://stevebuik.github.io/fork-malli-ideas/#!/basics#2020-07-1507:54steveb8nso now we have an example of keyword maps, validated by Malli but also transformed to string keys for Fork inputs#2020-07-1507:54steveb8nworks great!#2020-07-1508:16ikitommiadded the path->schema as malli.util/path-schemas, I think that is useful in things like generating forms from malli Schemas: you get an ordered list of value paths and the corresponding subschemas:
1. you can validate whole schema on blur / submitting
2. you can push subschema validation to components, faster local validation on keypress
works for all basic clojure data structures, also for nested sequences (via :malli.core/in marker in path)#2020-07-1508:17ikitommi#2020-07-1521:21rutledgepaulvThis sounds like it might help with something I've been pondering while working on https://github.com/RutledgePaulV/ui-kit! Still working out some of the state structure / data validation / conformance stuff before I go hard on nailing down all the component types#2020-07-1521:22rutledgepaulvTrying to do minimal redraws and making sense of reagent cursors + malli trees#2020-07-1608:55ikitommi👍#2020-07-1508:18ikitommie.g. just iterate the map entries and emit form components#2020-07-1508:31steveb8nthat’s an interesting idea. this could also be used with Fork to generate forms. might have to try that 🙂#2020-07-1508:32steveb8nhow far we’ve come from multi-tenant validation!#2020-07-1807:49ikitommifinal(?) cleanup for transformers: https://github.com/metosin/malli/pull/224#2020-07-1807:49ikitommibig change is that collection transformers don't coerce their type if children don't need transformation. In practise:
(m/decode [:vector keyword?] '("abba") (mt/json-transformer))
; => [:abba]
(m/decode [:vector string?] '("abba") (mt/json-transformer))
; => ("abba")#2020-07-1807:50ikitommifor string-transformer, works like before:
(m/decode [:vector string?] '("abba") (mt/string-transformer))
; => [:abba]#2020-07-1807:52ikitommi=> the transformer instances can decide how to transform the collections, with JSON, both Jsonista and Cheshire decode JSON arrays as vectors, so they don’t need any “just in case” transformations.#2020-07-1807:53ikitommiif you have a deep Malli Schema, which only has things that can be presented in JSON, the transformation engine return identity. Which is nice.#2020-07-1909:08mike_ananev@ikitommi Hi. Is there any roadmap for the first release? (How many features/errors should be improved/fixed before release? )#2020-07-1910:34ikitommi@mike1452 just the https://github.com/metosin/malli/pull/212 I think. But it seems to be a rabbit hole, can’t do the error thing in it elegantly with the current api, need either to discover a simple resolution for it, to change the explain (internal) api or the LensSchema api.#2020-07-1910:35ikitommialso, checked the regal-malli integration, and might do a quick re-pacakge of the protocols so that implementing a new schema instance doesn’t require requiring all malli namespaces.#2020-07-1910:37ikitommiall small things, the first release will be alpha with 98% stabile public api and 80% stabile internal. try to get all the breaking things (for the better) before the release.#2020-07-1911:57mike_ananev👍#2020-07-2308:22ikitommisci will be an optional dependency, why?
• faster standalone usage for JVM (2.5sec -> 0.5sec to load `malli.core`)
• smaller js-bundles: 120kb -> 7kb
first cut: https://github.com/metosin/malli/pull/227#2020-07-2308:43ikitommito make the sci-integration explicit, might be a good idea to make the whole default options customizable, instead of just the default registry. would allow one to pass sci as the default :evaluator , one could pass in default custom localizations for error messages etc. Wrote an issue here: https://github.com/metosin/malli/issues/228#2020-07-2308:44ikitommicomments welcome. it’s a rainy day, should have few hours later today to do that.#2020-07-2308:56plexusshould a schema-schema's properties delegate to the referenced schema?#2020-07-2308:56plexus(schema/properties (malli.core/-deref (schema/schema :ars/address)))
(schema/properties (schema/schema :ars/address))
#2020-07-2308:56plexusthe first returns a properties map, the second returns nil. would be nice if it actually returned the properties of the child schema#2020-07-2308:58plexus(I'm upgrading our project to the latest Malli. Nice addition of RefSchema and schema-schema!)#2020-07-2309:17plexusI guess the question is how transparent a RefSchema is expected to be. We had our implementation of a RefSchema before Malli added it, and in our case we tried to make it largely transparent, e.g. it implements MapSchema and LensSchema which delegate to the child schema. Was very convenient but maybe you prefer to be explicit and have (if (satisfies? RefSchema s) (-deref s) s) checks#2020-07-2309:20ikitommiGood question. :schema can now have it's own properties:
[::m/schema {:title "foo"} :ars/address]#2020-07-2309:22ikitommiwould help if there was distinction between properties and derived/accumulated properties#2020-07-2309:24ikitommiproperties are used in transforming schemas, e.g. into map-format. If they are looked from child, the map-format would duplicate the properties into ::m/schema#2020-07-2309:26ikitommia new Protocol / method would solve that, but not sure if that's the right fix#2020-07-2309:45plexusmakes sense to keep them separate if they have different uses, maybe some more helpers would already go a long way, there's no deref yet for instance, only -deref. A version of deref/resolve that checks for RefSchema and otherwise returns the schema directly would also be nice.#2020-07-2309:46plexuswhat about if you do an update-in, but one of the schemas on the path is not a map schema but a reference to a map schema? should that work? (we do stuff like that and it's quite handy :))#2020-07-2313:33ikitommiMerged the SCI Optional PR. From Changelog:
* 23.7.2020
* **BREAKING:**: `sci` is not a default dependency. Enabling sci-support:
* **Clojure**: add a dependency to `borkdude/sci`
* **ClojureScript**: also require `sci.core` (directly or via `:preloads`)#2020-07-2313:33ikitommibefore and after (a small sample app)#2020-07-2313:35borkdudeWhat is the total size of the app before and after, un-gzipped? What does optimized mean, Clojure advanced?#2020-07-2313:45ikitommihere’s the javascript:
➜ malli git:(master) ✗ ls -lh app2-sci
total 1536
-rw-r--r-- 1 tommi staff 8.8K Jul 23 16:43 app.js
-rw-r--r-- 1 tommi staff 177K Jul 23 16:43 cljs.js
-rw-r--r-- 1 tommi staff 16K Jul 23 16:43 malli.js
-rw-r--r-- 1 tommi staff 2.0K Jul 23 16:43 manifest.edn
-rw-r--r-- 1 tommi staff 556K Jul 23 16:43 sci.js
➜ malli git:(master) ✗ ls -lh app2
total 288
-rw-r--r-- 1 tommi staff 10K Jul 23 16:41 app.js
-rw-r--r-- 1 tommi staff 102K Jul 23 16:41 cljs.js
-rw-r--r-- 1 tommi staff 24K Jul 23 16:41 malli.js
-rw-r--r-- 1 tommi staff 987B Jul 23 16:41 manifest.edn
#2020-07-2313:46ikitommimanually zipped:
➜ malli git:(master) ✗ ls -lh app2-sci
total 344
-rw-r--r-- 1 tommi staff 2.3K Jul 23 16:43 app.js.gz
-rw-r--r-- 1 tommi staff 35K Jul 23 16:43 cljs.js.gz
-rw-r--r-- 1 tommi staff 4.7K Jul 23 16:43 malli.js.gz
-rw-r--r-- 1 tommi staff 624B Jul 23 16:43 manifest.edn.gz
-rw-r--r-- 1 tommi staff 119K Jul 23 16:43 sci.js.gz
➜ malli git:(master) ✗ ls -lh app2
total 80
-rw-r--r-- 1 tommi staff 3.0K Jul 23 16:41 app.js.gz
-rw-r--r-- 1 tommi staff 22K Jul 23 16:41 cljs.js.gz
-rw-r--r-- 1 tommi staff 6.5K Jul 23 16:41 malli.js.gz
-rw-r--r-- 1 tommi staff 370B Jul 23 16:41 manifest.edn.gz
#2020-07-2313:46ikitommithe app itself is silly, bare-bones malli with few schemas: https://github.com/metosin/malli/blob/master/app/malli/app2.cljc#2020-07-2313:47borkdudeok#2020-07-2313:47ikitommishadow config: https://github.com/metosin/malli/blob/master/shadow-cljs.edn#2020-07-2313:48borkdudemodules are a bit confusing since one thing can get randomly moved from one module to another#2020-07-2313:48borkdudebut overall, this is a win for people who don't want to use sci#2020-07-2313:52ikitommiso total sizes:
• js: 757,8kb -> 136kb
• gz: 161kb -> 31,5kb
, so 1/5 the size.#2020-07-2313:54ikitommiyup (javascript) is 223kB unpacked, so we can say malli can be small too. or batteries-included.#2020-07-2314:00ikitommianyone btw interested in writing a malli->typescript transformer? in a project with both, might be useful, most likely fun#2020-07-2314:01ikitommiwith yup, you can say:
type Person = yup.InferType<typeof personSchema>;
#2020-07-2314:02ikitommiI don’t think it would be much more work than the malli -> JSON Schema.#2020-07-2314:02ikitommimight be wrong 🙂#2020-07-2611:01mike_ananevHi! How I can express if A is optional in schema and if A exists then B should present too?#2020-07-2611:02borkdudeprobably using a predicate?#2020-07-2611:08mike_ananevI mean how to express it as spec (inside spec one value depends on another) ?#2020-07-2611:12mike_ananev{
:java-src-folder "test-projects/javac/src/java"
:javac-options ["-target" "1.8" "-source" "1.8" "-Xlint:-options"] }
How to express as spec if :java-src-folder is present then :javac-options should be present?#2020-07-2611:57ikitommi@mike1452 you can compose rules with :and, from README:
`(-> [:and [:map
[:password string?]
[:password2 string?]]
[:fn {:error/message "passwords don't match"
:error/path [:password2]}
'(fn [{:keys [password password2]}]
(= password password2))]]
(m/explain {:password "secret"
:password2 "faarao"})
(me/humanize))
; {:password2 ["passwords don't match"]}`#2020-07-2612:01mike_ananev@ikitommi Thanks! 👍#2020-07-2709:04steveb8nQ: I’m having trouble with Cursive and shadow compilation ever since I updated to the dynaload/sci update. anyone else seeing weird behaviour?#2020-07-2709:06ikitommi@steveb8n do you have the sci.core preloaded?#2020-07-2709:07borkdudeAre you using sci?#2020-07-2709:07steveb8nno but I don’t want sci in the build#2020-07-2709:07steveb8nno, not using sci#2020-07-2709:07borkdudeThen you should not preload it and it should all work fine theoretically.#2020-07-2709:07steveb8nI did previously but I don’t need it (yet) so would rather save the bundle output size#2020-07-2709:07ikitommiwhat kind of errors do you get?#2020-07-2709:08steveb8n➜ client-editor git:(master) ✗ shadow-cljs -A:dev --verbose release frontend
shadow-cljs - config: /Users/steve/Documents/dev-personal/nextdoc-cloud/client-editor/shadow-cljs.edn
shadow-cljs - starting via “clojure”
[:frontend] Compiling ...
-> build target: :browser stage: :configure
<- build target: :browser stage: :configure (6 ms)
-> Resolving Module: :main
The required namespace “borkdude.dynaload-cljs” is not available, it was required by “malli/sci.cljc”.
➜ client-editor git:(master) ✗ clojure -Stree
org.clojure/clojure 1.10.1
org.clojure/core.specs.alpha 0.2.44
org.clojure/spec.alpha 0.2.176
cljs-http/cljs-http 0.1.46
noencore/noencore 0.3.4
commons-codec/commons-codec 1.11
org.clojure/tools.namespace 0.2.11
org.clojure/core.async 0.4.474
org.clojure/tools.analyzer.jvm 0.7.0
org.clojure/tools.analyzer 0.6.9
org.clojure/core.memoize 0.5.9
org.clojure/core.cache 0.6.5
org.clojure/data.priority-map 0.0.7
org.ow2.asm/asm-all 4.2
com.lucasbradstreet/cljs-uuid-utils 1.0.2
riverford/compound 2020.01.09
appliedscience/js-interop 0.1.19
org.clojure/clojurescript 1.10.520
org.clojure/data.json 0.2.6
org.clojure/google-closure-library 0.0-20170809-b9c14c6b
org.clojure/google-closure-library-third-party 0.0-20170809-b9c14c6b
org.mozilla/rhino 1.7R5
com.google.javascript/closure-compiler-unshaded v20180805
com.google.jsinterop/jsinterop-annotations 1.0.0
com.google.javascript/closure-compiler-externs v20180805
com.google.guava/guava 25.1-jre
com.google.errorprone/error_prone_annotations 2.1.3
org.codehaus.mojo/animal-sniffer-annotations 1.14
com.google.j2objc/j2objc-annotations 1.1
org.checkerframework/checker-qual 2.0.0
com.google.code.findbugs/jsr305 3.0.2
args4j/args4j 2.0.26
com.google.protobuf/protobuf-java 3.0.2
com.google.code.gson/gson 2.7
com.cognitect/transit-cljs 0.8.256
com.cognitect/transit-js 0.8.846
fork/fork https://github.com/luciodale/fork.git 25276c0
grafeo/grafeo 0.1.4
r0man/alumbra.printer 0.1.1
r0man/alumbra.js 0.1.0
clj-http/clj-http 3.9.1
org.apache.httpcomponents/httpasyncclient 4.1.3
org.apache.httpcomponents/httpcore-nio 4.4.6
slingshot/slingshot 0.12.2
commons-io/commons-io 2.6
org.apache.httpcomponents/httpcore 4.4.9
org.apache.httpcomponents/httpclient-cache 4.5.5
org.apache.httpcomponents/httpclient 4.5.5
commons-logging/commons-logging 1.2
potemkin/potemkin 0.4.5
clj-tuple/clj-tuple 0.2.2
org.apache.httpcomponents/httpmime 4.5.5
cheshire/cheshire 5.8.1
com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.9.6
tigris/tigris 0.1.1
com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.9.6
re-frame/re-frame 0.10.6
org.clojure/tools.logging 0.3.1
net.cgrand/macrovich 0.2.0
reagent/reagent 0.7.0
cljsjs/react-dom 15.5.4-0
cljsjs/react 15.5.4-0
cljsjs/react-dom-server 15.5.4-0
cljsjs/create-react-class 15.5.3-0
cljc.java-time/cljc.java-time 0.1.11
cljs.java-time/cljs.java-time 0.1.16
henryw374/js-joda 1.12.0-1
metosin/malli https://github.com/metosin/malli.git 3a670d8
com.gfredericks/test.chuck 0.2.10
instaparse/instaparse 1.3.6
com.andrewmcveigh/cljs-time 0.5.1
borkdude/dynaload https://github.com/borkdude/dynaload.git 52f71bc
borkdude/edamame 0.0.11-alpha.12
org.clojure/tools.reader 1.3.2
org.clojure/test.check 1.0.0
funcool/promesa 4.0.2
com.rpl/specter 1.1.3-SNAPSHOT
riddley/riddley 0.1.12
metosin/reitit-frontend 0.2.9
metosin/reitit-core 0.2.9
meta-merge/meta-merge 1.0.0
metosin/reitit 0.2.9
metosin/reitit-middleware 0.2.9
metosin/muuntaja 0.6.1
com.cognitect/transit-clj 0.8.313
com.cognitect/transit-java 0.8.337
org.msgpack/msgpack 0.6.12
com.googlecode.json-simple/json-simple 1.1.1
org.javassist/javassist 3.18.1-GA
javax.xml.bind/jaxb-api 2.3.0
lambdaisland/deep-diff 0.0-25
mvxcvi/puget 1.0.3
fipp/fipp 0.6.14
mvxcvi/arrangement 1.1.1
org.clojure/core.rrb-vector 0.0.13
tech.droit/clj-diff 1.0.0
metosin/reitit-swagger 0.2.9
metosin/reitit-ring 0.2.9
ring/ring-core 1.7.1
commons-fileupload/commons-fileupload 1.3.3
clj-time/clj-time 0.14.3
joda-time/joda-time 2.9.9
crypto-random/crypto-random 1.2.0
ring/ring-codec 1.1.1
crypto-equality/crypto-equality 1.0.0
metosin/reitit-schema 0.2.9
metosin/schema-tools 0.10.5
prismatic/schema 1.1.9
metosin/reitit-sieppari 0.2.9
metosin/sieppari 0.0.0-alpha6
metosin/reitit-http 0.2.9
metosin/reitit-spec 0.2.9
metosin/spec-tools 0.8.2
com.fasterxml.jackson.core/jackson-databind 2.9.7
com.fasterxml.jackson.core/jackson-core 2.9.7
com.fasterxml.jackson.core/jackson-annotations 2.9.0
metosin/reitit-swagger-ui 0.2.9
metosin/ring-swagger-ui 2.2.10
metosin/jsonista 0.2.2
org.ow2.asm/asm 5.1
virgil/virgil 0.1.6
com.fasterxml.jackson.datatype/jackson-datatype-jsr310 2.9.7
metosin/reitit-interceptors 0.2.9
camel-snake-kebab/camel-snake-kebab 0.4.0
➜ client-editor git:(master) ✗#2020-07-2709:09steveb8nI can also make it fail by starting a clojure repl in the terminal and trying to require dynaload#2020-07-2709:09borkdudemaybe kill your ~/.gitlibs ?#2020-07-2709:10borkdudeand clear out your .cpcache#2020-07-2709:10borkdudeand use -Sforce#2020-07-2709:10steveb8nClojure 1.10.1or git:(master) ✗ clojure
user=> (require ’borkdude.dynaload-clj)
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate borkdude/dynaload_clj__init.class, borkdude/dynaload_clj.clj or borkdude/dynaload_clj.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.#2020-07-2709:11steveb8nI did try deleting it from gitlibs but I haven’t tried deleting the whole dir#2020-07-2709:11steveb8nhave already deleted .cpcache#2020-07-2709:11steveb8ntrying -Sforce ….#2020-07-2709:12steveb8nclojure -Sforce
Clojure 1.10.1
user=> (require ’borkdude.dynaload-clj)
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate borkdude/dynaload_clj__init.class, borkdude/dynaload_clj.clj or borkdude/dynaload_clj.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.#2020-07-2709:12borkdude@steveb8n when you do clojure -Spath do you see borkdude/dynaload there?#2020-07-2709:12steveb8nwhat’s even stranger is that it works in another project and in CI for this (problematic) project#2020-07-2709:13borkdudemaybe it's a shadow problem? can you try with vanilla CLJS?#2020-07-2709:13steveb8nyes. with -Spath it is present#2020-07-2709:13borkdudeoh you tried already with the JVM huh. that's strange#2020-07-2709:13steveb8nthat’s why I also tested using the clj repl. to exclude shadow#2020-07-2709:14steveb8nCursive is also weird. it appears in the deps panel but not in “external libraries”#2020-07-2709:14ikitommi➜ ~ clojure -Sdeps '{:deps {metosin/malli {:git/url "" :sha "627b0f0592d129a317f8ccf3dff5296376948bf9"}}}'
Checking out: at 627b0f0592d129a317f8ccf3dff5296376948bf9
Clojure 1.10.1
user=> (require 'borkdude.dynaload-clj)
Syntax error (ClassNotFoundException) compiling at (REPL:1:1).
'borkdude.dynaload-clj#2020-07-2709:15borkdudewhat SHA is malli using for dynaload? maybe it was a branch that got deleted?#2020-07-2709:15borkdudethe latest SHA is 52f71bc2cb7389a932835fe02f185e3801f7e063#2020-07-2709:16steveb8nmalli is using 52….#2020-07-2709:16steveb8n@ikitommi when I run that in my terminal, it works ok#2020-07-2709:16steveb8nsorry. wrong#2020-07-2709:16steveb8nrepl starts but require fails#2020-07-2709:17steveb8nif you try the same require with the 52… sha, does it work?#2020-07-2709:17borkdude$ clojure -Sdeps '{:deps {metosin/malli {:git/url "" :sha "627b0f0592d129a317f8ccf3dff5296376948bf9"}}}'
Cloning:
Checking out: at 627b0f0592d129a317f8ccf3dff5296376948bf9
Cloning:
Checking out: at 52f71bc2cb7389a932835fe02f185e3801f7e063
Clojure 1.10.1
user=> (require 'borkdude.dynaload-clj)
nil
user=>#2020-07-2709:17borkdudeworks fine over here#2020-07-2709:17borkdudedoing this from a clean dir#2020-07-2709:18steveb8nclojure -Sdeps ‘{:deps {metosin/malli {:git/url “https://github.com/metosin/malli.git” :sha “627b0f0592d129a317f8ccf3dff5296376948bf9"}}}’
Clojure 1.10.1
user=>
user=>
user=> (require ’borkdude.dynaload-clj)
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate borkdude/dynaload_clj__init.class, borkdude/dynaload_clj.clj or borkdude/dynaload_clj.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.#2020-07-2709:19steveb8nmust be a deps boycott on Australia#2020-07-2709:19ikitommiworking here now too :thinking_face:#2020-07-2709:19ikitommichecked out dynaload manually, started working after that.#2020-07-2709:19borkdudemaybe it's a deps.edn bug?#2020-07-2709:19steveb8nyeah I could try that#2020-07-2709:20steveb8nexplicit checkout locally#2020-07-2709:20borkdude$ clojure -Sdescribe
{:version "1.10.1.536"#2020-07-2709:20ikitommiborkdude/dynaload {:git/url ""
:sha "52f71bc2cb7389a932835fe02f185e3801f7e063"}#2020-07-2709:21steveb8nI’ve got 1.10.1.447#2020-07-2709:21steveb8nI’ve seen deps bugs before. I might try updating clojure cli#2020-07-2709:22borkdudeI'm willing to cut a .jar release later on, but since malli itself is still git only, I wasn't in a hurry#2020-07-2709:25steveb8nI tried adding an explicit dep on dynaload. same behaviour as with the transitive dep via malli#2020-07-2709:26steveb8nstill waiting on brew upgrade 🙂#2020-07-2709:28steveb8nare either of you using Cursive? if so, do you see dynaload in the “External Libraries”?#2020-07-2709:30steveb8nafter the clojure deps upgrade, the require works in the terminal. 1 step forward!#2020-07-2709:32steveb8nshadow compile from cli also now works. it is looking like a deps bug#2020-07-2709:32steveb8ntrying Cursive ….#2020-07-2709:34steveb8ncursive/shadow via repl still broken. I’ll keep digging…#2020-07-2709:38borkdudemaybe cursive uses its own version of deps.edn or maybe tools.deps directly which may be behind#2020-07-2709:40steveb8nthat’s exactly what is was! I switched (in prefs) to using the CLI dep executable and now shadow works inside Cursive as well#2020-07-2709:40steveb8nI got bitten by a double-whammy: deps bug and Cursive config#2020-07-2709:41steveb8nthanks for the help team 🙏#2020-07-2709:48ikitommiwhat was the root cause for the bug? I’m using 1.10.1.536 and still saw it. A stale cache?#2020-07-2709:57borkdudemaybe in your case, an AOT-cache related bug?#2020-07-2709:58steveb8nnot sure as I’ve blown away so much stuff now#2020-07-2709:58steveb8nI’m now on 1.10.1.561#2020-07-2709:59borkdudeI mean in ikitommi's case, since he had a ClassNotFoundException#2020-07-2709:59borkdudewhereas steveb8n got a file not found related exception#2020-07-2710:01steveb8nmy best guess is a bug in handling transitive deps via git deps#2020-07-2710:02steveb8nbtw: loving the reduction in bundle size. thanks both of you#2020-07-2710:02steveb8nalthough I then added vega lite and added another 200k#2020-07-2810:49dharriganAny thoughts on my PR? https://github.com/metosin/malli/pull/234. It's a shorthand way of fixing the length of the string to be accepted 🙂#2020-07-2912:48ikitommi@dharrigan not fond of that. You can already say [:string {:min 4, :max 4}] , which is not that much longer than [:string {:length 4}]. And you can always uses aliases to save on typing:
[:schema
{:registry {::str4 [:string {:min 4, :max 4}]}}
::str4]
or:
(def string4 [:string {:min 4, :max 4}])
(m/validate string4 "kikka")
; => false#2020-07-2912:49ikitommithere could be a pretty error formatter for that, e.g.
(and min (= min max)) (str "should be " min " characters")
#2020-07-2912:49ikitommi=>
(-> [:schema {:registry {::str4 [:string {:min 4, :max 4}]}} ::str4]
(m/explain "kikka")
(me/humanize))
; => ["should be 4 characters"]#2020-07-2913:00ikitommifixed that in master, thanks for giving it a though.#2020-07-2913:02ikitommiJSON Schema also has only minLength & maxLength.#2020-07-2913:09ikitommi(def registry
(reduce
(fn [acc i]
(assoc acc (str "string" i) [:string {:min i, :max i}]))
{}
(range 5)))
registry
;{"string0" [:string {:min 0, :max 0}],
; "string1" [:string {:min 1, :max 1}],
; "string2" [:string {:min 2, :max 2}],
; "string3" [:string {:min 3, :max 3}],
; "string4" [:string {:min 4, :max 4}]}
(-> [:map {:registry registry}
[:s1 "string1"]
[:s2 "string2"]
[:s3 "string3"]]
(m/explain {:s1 "hi", :s2 "hi", :s3 "hi"})
(me/humanize))
;{:s1 ["should be 1 characters"]
; :s3 ["should be 3 characters"]}#2020-07-2915:41ikitommiIs anyone leaning on the current format of :errors of m/explain ?#2020-07-2915:43ikitommiAbout to break the :path and :in so that :path is a valid pointer to mu/get-in.#2020-07-2915:44ikitommithis gives symmetry, that should have been there since beginning.#2020-07-2915:57ikitommithe paths are so much better now 🙂
(m/explain
[:multi {:dispatch :type}
[:sized [:map
[:type [:= :sized]]
[:size [:maybe int?]]]]
[:human [:map
[:type [:= :human]]
[:name string?]
[:age int?]
[:address [:maybe [:map [:country keyword?]]]]]]]
{:type :human
:name "inkeri"
:age "100"
:address {:country "sweden"}})
;{:schema [:multi
; {:dispatch :type}
; [:sized [:map [:type [:= :sized]] [:size [:maybe int?]]]]
; [:human [:map [:type [:= :human]] [:name string?] [:age int?] [:address [:maybe [:map [:country keyword?]]]]]]],
; :value {:type :human, :name "inkeri", :age "100", :address {:country "sweden"}},
; :errors (#Error{:path [:human :age]
; :in [:age]
; :schema int?
; :value "100"}
; #Error{:path [:human :address 0 :country]
; :in [:address :country]
; :schema keyword?
; :value "sweden"})}#2020-07-2915:58ikitommi(Schemas with named branches, e.g. :map and :multi use the name instead of vector index.#2020-07-2923:12danielglauserHow can you validate a "naked" series of integers? Like 1,4,8,10#2020-07-2923:13danielglauserNot in a vector, they are coming in as a query parameter.#2020-07-3007:07ikitommias one string? Maybe something like:
(m/decode
[:vector {:decode/string (partial str/split ",")} int?]
"1,2,4,8,10"
mt/string-transformer)
not near a computer, so not 100% sure it works#2020-07-3007:08ikitommifirst splits the string, the decodes parts string->int#2020-07-3016:07danielglauserThanks, I'll give that a try. I came up with this but your solution seems simpler:
(defn ids?
"Takes in a string and returns true if that string is of the form
1,4,7,9 or a sequence (in the math sense) of positive integers."
[data]
(let [ids (-> data
(clojure.string/split #",")
(as-> ids (mapv #(Integer/parseInt %) ids)))]
(every? pos? ids)))
(def GetClassesQueryOptions
[:map
[:approved {:optional true} boolean?]
[:instructors {:optional true} [:fn (fn [ids] (ids? ids))]]
[:machine {:optional true} [:fn (fn [ids] (ids? ids))]]
[:music {:optional true} [:fn (fn [ids] (ids? ids))]]
[:bookmarked {:optional true} boolean?]
[:minlength {:optional true} int?]
[:maxlength {:optional true} pos?]])#2020-07-3017:20ikitommiIt's good to keep decoding and validation in separate steps. @alexmiller has many times called the spec conform "a meat grinder", as it bundles the two and runs the transformations every time.#2020-07-3116:32ikitommiAfter #238, there will be two ways to program with schemas: via schema keys (`:in` in explain) and value keys (`:path`in explain). e.g. one can walk easily over parts of schemas that don’t effect the value paths:
(def Schema
(m/schema
[:maybe
[:and
[:map
[:id string?]
[:tags [:set keyword?]]
[:address
[:and
[:map
[:street {:optional true} string?]
[:lonlat {:optional true} [:tuple double? double?]]]
[:fn '(fn [{:keys [street lonlat]}] (or street lonlat))]]]]
[:fn '(fn [{:keys [id tags]}] (and id tags))]]]))
(mu/get-in Schema [0 0 :address 0 :lonlat])
; => [:tuple double? double?]
(mu/get-in* Schema [:address :lonlat])
; => [:tuple double? double?]#2020-08-0216:38ikitommiOh my. Accumulating both :in and :path in explain errors seems redundant. If we know Schema and either of schema-path (`:path`) or a value-path (`:in`), we can calculate the other. Simplifies both -explain and the malli utils. Sweet.#2020-08-0216:39ikitommiand the code is dead simple:
(defn in->path [schema in]
(loop [i 0, s schema, acc []]
(or (and (>= i (count in)) acc)
(recur (inc i) (get s (in i)) (if-not (m/-key s) (conj acc (in i)) acc)))))
(defn path->in [schema path]
(loop [i 0, s schema, acc []]
(or (and (>= i (count path)) acc)
(let [[i k] (if-let [k (m/-key s)] [i k] [(inc i) (path i)])]
(recur i (get s k) (conj acc k))))))#2020-08-0216:40ikitommi(also, performant, as it’s only protocols and vector indexes)#2020-08-0219:18borkdude@ikitommi Published dynaload to Clojars: https://clojars.org/borkdude/dynaload#2020-08-0308:00ikitommithanks! Will update the deps#2020-08-0411:35ikitommimerged #238 in master. The Breaking:
• :path in explain is re-implemented: map keys by value, others by child index
• m/-walk and `m/Walker` uses `:path`, not `:in`
• m/-outer has new parameter order: `walker schema path children options`
• malli.util/path-schemas replaced with `malli.util/subschemas` & `malli.util/distict-by`
• LensSchema has a new `-keep` method
• renamed some non-user apis in `malli.core` & `malli.util`
• moved map-syntax helpers from `malli.core` to `malli.util`
• dynaload `com.gfredericks/test.chuck`#2020-08-0411:38ikitommiThere are few more PRs to go, many half-way complete and should unroll after the 238. But, going back to work within few days, don’t know when will have time to finish all the things.#2020-08-0411:46ikitommiApplied for Malli for the Clojurists Together aug-sep, but haven’t heard anything, so just:crossed_fingers:.#2020-08-0510:50ikitommiComments on https://github.com/metosin/malli/pull/194? e.g. allowing spec2-like schema+select using just :map.#2020-08-0510:51ikitommi#2020-08-0514:50teodorluHey! I just found :map-of through after some searching, first reading the Readme, then guessing its name after rechecking the documentation for Spec. Is there interest for a PR adding an example to the Readme?#2020-08-0515:56ikitommidefinetely! All doc improvements most welcome#2020-08-0515:57ikitommisomeone suggested setting up real docs too, under docs so that cljdoc would pick them up too#2020-08-0614:39ikitommiwith current master:#2020-08-0614:39ikitommi#2020-08-0708:46borkdudeHoe does one combine validate and transform? E.g. this doesn't crash:
(prn (m/decode int? :foo mt/string-transformer))
I'm not saying it should, just wondering how to do it. Not clear from the README#2020-08-0708:48borkdudeShould I first call m/valid, if not valid, then m/explain and else m/decode, effectively traversing the structure twice?#2020-08-0709:21ikitommiIf you need decoding, the flow should be:
1. decode
2. validate
3. explain on error#2020-08-0709:22ikitommithere is 2-3 walks in the error case.#2020-08-0709:22ikitommifor happy case, 1-2#2020-08-0709:24ikitommihaving a seoarate optimized validate makes the happy case fast#2020-08-0709:24ikitommithe docs could have examples on this...#2020-08-0709:26ikitommithe m/decoder doesn't have to walk the structure, it returns an function to transform just the parts that need to be decoded. In case there is nothing to do, it returns identity#2020-08-0709:35borkdude@ikitommi The concrete example I was going to try:
$ cat deps.edn
{:deps {metosin/malli {:git/url "" :sha "2bd749f7148e28a379f1e628a32188e7f6cf0bc4"}
borkdude/edamame {:git/url "" :sha "64c7eb43950eb500ba7429dded48257cd15355ae"}}}
$ cat src/edamalli/core.clj
(ns edamalli.core
(:require [edamame.core :as e]
[malli.core :as m]
[malli.transform :as mt]))
(defrecord WrappedNum [obj loc])
(defn postprocess [{:keys [:obj :loc]}]
(if (number? obj) (->WrappedNum obj loc) obj))
(defn -main [& args]
(prn (e/parse-string "[:foo 42]" {:postprocess postprocess}))
;; TODO:
;; - validate that the WrappedNum contains value < 42
;; - then transform it to only that number
;; - else raise error, printing the location metadata of that number
)
$ clojure -m edamalli.core
[:foo #edamalli.core.WrappedNum{:obj 42, :loc {:row 1, :col 7, :end-row 1, :end-col 9}}]#2020-08-0711:06zclj@ikitommi I am parsing a schema into a malli-schema. The original might contain recursive references to other "entities" in the schema. I do not know this up front. Are there any trade-offs in putting all my potential recursive entity references in a [:ref ], even if they turn out not to be?#2020-08-0711:44ikitommi@borkdude would [:foo 42] be transformed to [:foo 42] , as would [:foo :bar #{42}] to itself and {:a 41} would fail on the fact that there was a number that was not 42?#2020-08-0711:45ikitommihappy to help, sample inputs -> outputs would help.#2020-08-0711:45borkdude@ikitommi No, [:foo (WrappedNum. x y)] would be transformed to [:foo x] only if x < 42, else error with explain using y#2020-08-0711:46ikitommicould that validation happen already in the :postprocess?#2020-08-0711:47borkdudeI just want to feed this data to malli and not intertwine parsing data from text to sexprs with malli validation#2020-08-0711:48ikitommiok. can the input be anything, e.g [:foo {:bar (WrappedNum. x y)}]?#2020-08-0711:48borkdude[:foo y] could also be {:foo y} if that's easier for you#2020-08-0711:49borkdudeNo, it's more like person: {:name "foo" :age 42}, let's say#2020-08-0711:49borkdudeso attribute+value#2020-08-0711:50borkdudeso unwrapping would just be (:obj wrapped), that would be the transform step#2020-08-0711:50borkdudeThe use case for this is: normally edamame doesn't let you have location metadata for numbers and strings, but using a wrapped value you can have that#2020-08-0711:51borkdudeso I want to use malli like normally, but just use the location metadata in the wrapped value for reporting errors and discard it if the value is valid#2020-08-0711:59borkdudeI believe in spec you would write a conformer for this#2020-08-0712:12ikitommiyes, this is kinda tricky with current malli, as there is not yet a parsing api, like conform.#2020-08-0712:13ikitommialso, there is no custom overridable validator, so one needs to describe the given data structure (here, a tuple with keyword and a record).#2020-08-0712:15borkdudeok, so one would write a schema using the records and if everything's ok, then postwalk yourself, unwrapping them?#2020-08-0712:15ikitommibut, something like this:
;; schemas
(def <42 [:and int? [:< 42]])
(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure :loc} [:obj <42]]])
;; validator and encoders for both success & failure
(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))
;; in action
(defn parse-validate-and-transform [s]
(let [x (e/parse-string s {:postprocess postprocess})]
(if (valid? x) (success x) (failure x))))
=>
(parse-validate-and-transform "[:foo 41]")
; => [:foo 41]
(parse-validate-and-transform "[:foo 42]")
; => [:foo {:row 1, :col 7, :end-row 1, :end-col 9}]#2020-08-0712:16ikitommiyes, postwalk would do. or a recursive schema definition. if the wrapped records can be anywhere#2020-08-0712:16borkdudelet me try your snippet#2020-08-0712:17borkdude$ clojure -m edamalli.core
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:76).
:sci-not-available {:code ":obj"}
That was unexpected, I don't need sci in this example?#2020-08-0712:18ikitommioh, should not.#2020-08-0712:19ikitommi(def Schema [:tuple keyword? [:map {:encode/success (fn [x] (:obj x)), :encode/failure (fn [x] (:loc x)) [:obj <42]}]])#2020-08-0712:21ikitommipushed e19872273c3660fbc482dcff4c2d8439dbcbb2a6, which should allow naked keywords as functions.#2020-08-0712:21ikitommi(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure :loc} [:obj <42]]])#2020-08-0712:21borkdude#2020-08-0712:22borkdudeI'm still getting the sci-not-available error#2020-08-0712:23borkdudeWhen I do include sci, I get:
[:foo {:row 1, :col 7, :end-row 1, :end-col 9}]#2020-08-0712:24borkdudeI guess I should throw my own error in :encode/failure?#2020-08-0712:26ikitommiyes, that’s one place to do it. will check why sci is needed. just a sec.#2020-08-0712:27borkdude(defn parse-validate-and-transform [s]
(let [x (e/parse-string s {:postprocess postprocess})]
(if (valid? x)
(prn "SUCCESS" (success x))
(prn "ERROR" (failure x)))))
$ clojure -m edamalli.core
"ERROR" [:foo {:row 1, :col 7, :end-row 1, :end-col 9}]#2020-08-0712:29borkdudehaha, when I do this:
:encode/failure {:message "should be lower than 42"}
I get:
$ clojure -m edamalli.core
Execution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:54).
Could not resolve symbol: should [at line 1, column 1]
#2020-08-0712:29borkdudeI have no idea what I'm doing, since I don't know these APIs well. I'll take a look after work again some time#2020-08-0712:36ikitommi8e067b3d004d1692cbfc695bc73d7e032ecb6e7f#2020-08-0712:37ikitommithe code used sci for all non fn?s, no to all non ifn?s. need to add tests.#2020-08-0712:38borkdude@ikitommi I now have this:
(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure {:message "should be lower than 42"}} [:obj <42]]])
Output:
"eval!" "should be lower than 42"
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:76).
:sci-not-available {:code "should be lower than 42"}#2020-08-0712:39borkdudeI might be doing something wrong, but it seems there's a debug println in there?#2020-08-0712:43ikitommipicard-facepalm my bad. but this kinda works (but is bad, should be better when we have the parsing api):#2020-08-0712:43ikitommi➜ ~ clojure -Sdeps '{:deps {metosin/malli {:sha "230b1767729aad3e02568f1320855e2b45d2d9b5", :git/url ""}, borkdude/edamame {:sha "64c7eb43950eb500ba7429dded48257cd15355ae", :git/url ""}}}'
Checking out: at 230b1767729aad3e02568f1320855e2b45d2d9b5
Clojure 1.10.1
user=> (ns edamalli.core
(:require [edamame.core :as e]
[malli.core :as m]
[malli.transform :as mt]))
(defrecord WrappedNum [obj loc])
(defn postprocess [{:keys [:obj :loc]}]
(if (number? obj) (->WrappedNum obj loc) obj))
(defn fail! [{:keys [:obj :loc]}]
(throw (ex-info (str "so bad " obj "/" loc) {})))
(def <42 [:and int? [:< 42]])
(def Schema [:tuple keyword? [:map {:encode/success :obj,
:encode/failure fail!} [:obj <42]]])
(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))
(defn parse-validate-and-transform [s]
(let [x (e/parse-string s {:postprocess postprocess})]
(if (valid? x) (success x) (failure x))))
edamalli.core=> (parse-validate-and-transform "[:foo 41]")
[:foo 41]
edamalli.core=> (parse-validate-and-transform "[:foo 42]")
Execution error (ExceptionInfo) at edamalli.core/fail! (REPL:2).
so bad 42/{:row 1, :col 7, :end-row 1, :end-col 9}#2020-08-0712:45ikitommi@zclj there is a small (have not measured) penalty for using ref-schemas, one function hop basically as the values are memoized.#2020-08-0713:03zcljok, that's fine since I have to do something to solve it anyway, by post-walking or such. Doing it up-front with malli ref considerable make the design simpler. Thanks for the info!#2020-08-0717:50ikitommirollback on the perf info @zclj . Validation & explain perf is about the same but transforming values is potentially much slower. Why? Malli can't optimize over :refs. Transformation behind :ref could be no-op, but it won't be removed as it's wrapped in a function. Still, orders of magnitude faster than with spec(-tools)#2020-08-0809:45zcljthanks for the update! In my use-case I will also do generation from the schema, where I will blow the stack if I don't use :ref for recursive references. Are there any implications for using :ref for potentially non-recursive entities in that case?#2020-08-0712:45borkdudeworks, thanks#2020-08-0717:50ikitommirollback on the perf info @zclj . Validation & explain perf is about the same but transforming values is potentially much slower. Why? Malli can't optimize over :refs. Transformation behind :ref could be no-op, but it won't be removed as it's wrapped in a function. Still, orders of magnitude faster than with spec(-tools)#2020-08-1012:02Shuai LinHow can I express a schema of "map of string keys to integer values?" in malli?#2020-08-1012:03ikitommi@linshuai2012 [:map-of string? int?]#2020-08-1012:03Shuai LinThanks @ikitommi, just found it's there in README, I read through the README and missed it 😄#2020-08-1012:08Shuai Linbtw thanks for creating malli, it's much data friendly and intuitive than spec ! I'm most looking forward to see the fdef equivalent https://github.com/metosin/malli/issues/125#2020-08-1012:08ikitommime too 😉#2020-08-1012:19borkdudeI'm also looking forward to a s/conformer equivalent#2020-08-1013:24zclj@ikitommi I have a schema where some maps have a lot of optional keys, up to 50-ish. When I generate from this schema I got a, to me, surprising result. What happens is that even for very small sizes I get very large generated maps (and since they are recursive this is magnified). Looking into how malli generates maps with optional keys, it seems that each key will have a 50% change of being included, so there is no notion of 'size' for the complete map. This also effect shrinking. So in practice my first generated map can be empty, while the second sample contains 20 keys. Intuitively I would expect the generation of optional keys to start small and then grow larger with size. Is this something you think malli should take care of or is it out of scope for malli and something I need to take care of with custom generators?#2020-08-1013:28ikitommi@zclj if you have a simple improved algorithm for generating with optional keys, please PR, the code is kinda naive atm: https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L84-L87#2020-08-1308:11zclj@ikitommi I have tried out some different solutions to this and though about it. My conclusion so far is that there are really no perfect default solution. As is usally the case in software testing (where I use the generators) the proper distribution of values are dependent on both the system under test and the outcome you want (positive tests, negative tests etc.) With this in mind, would you be open to a solution where malli could allow for the options to contain a :malli.generators/map-gen-optional-fn, where the user can provide their own implementaiton of the value selection and keep the default as is? In general, if malli provided such options it would be an awesome way to experiment with different distributions and allow for libraries to provide commonly used variants.#2020-08-1308:15ikitommiwith the current api, you can override the whole :map generator with:
(defmethod malli.generator/-schema-generator :map [schema options] (-my-map-gen schema options))
#2020-08-1308:16ikitommiwould that be enough?#2020-08-1308:17ikitommi(not happy that one can globally override the generators, I concider this as mutable evil, but is like that today)#2020-08-1308:29zcljI experimented some with that option but did not pursue it further due to me still wanting the -recur funtionallity and then having to do that in my own code. I could of course re-use mallis -recur but it felt that I started to depend to much on malli internals for this to be a clean method of extension, but maybe I was to defensive in that decision?#2020-08-1308:41ikitommineed to publish a malli internal naming convention guide. my idea was that all public - starting functions are public for library extensions / advanced use, tracked separately in CHANGELOG from the public end user api. Using those for basic use cases is a sign of invalid usage. Those will break more than the user-facing public api, but so that it’s easy for advanced users adapt to changes, e.g. no silent breaking changes after initial public release.#2020-08-1308:42ikitommibut, on second thought, if that one new option is good / you can provide alternative example impl (into README/docs) that might be of value to someone else, open to PR for adding that.#2020-08-1308:43ikitommias options are passed in to -map-gen already, it’s most likely <4 lines of code + tests + docs.#2020-08-1308:43ikitommi(and doesn’t add much to cljs bundle size)#2020-08-1413:27eskosIs there a reason the private stuff simply isn’t defn- ? Are the private parts split across multiple namespaces in such a way it would make this complicated?#2020-08-1413:51ikitommireal privates are ^:private. Vars like malli.core/-map-schema are public but only needed if you build your own registry, which is for advanced users only. Another is malli.core/-parse-entries which is a helper for building your own Schema instance, which wants to use the map-syntax#2020-08-1413:52ikitommispec is mostly closed for extensions, malli is built to be extended.#2020-08-1013:32zclj@ikitommi I will take a look and see if I can figure something out 🙂#2020-08-1014:01Shuai Linanother question, can I add some meta information to each field? Like this:
(def Person
[:map
[:name string? {:doc "The name of the person"}]
[:age string?]])#2020-08-1014:11ikitommi@linshuai2012 map entries can have optional property map:
(def Person
[:map
[:name {:doc "The name of the person"} string?]
[:age string?]])#2020-08-1014:12ikitommijust polishing the generic traversal of child-parents:
(defn parent-properties [schema path]
(loop [i (count path), acc []]
(if (>= i 0)
(let [p (subvec path 0 i)
s (mu/get-in schema p)
++ #(conj % (m/properties s))]
(if (and (m/entries s) (< i (count path)))
(recur (dec i) (++ (conj acc (m/properties (mu/get-in schema (conj p [::m/entry (path i)]))))))
(recur (dec i) (++ acc))))
acc)))
(parent-properties
[:map {:error/message "1"}
[:y {:error/message "2"}
[:and {:error/message "3"}
[:map {:error/message "4"}
[:x {:error/message "5"}
[:and {:error/message "6"}
int? [:> {:error/message "7"} 18]]]]]]]
[:y 0 :x 1])
;[#:error{:message "7"}
; #:error{:message "6"}
; #:error{:message "5"}
; #:error{:message "4"}
; #:error{:message "3"}
; #:error{:message "2"}
; #:error{:message "1"}]#2020-08-1016:19ikitommi@borkdude collected some thoughts on the s/conform here: https://github.com/metosin/malli/issues/241. Comments and ideas welcome.#2020-08-1017:00borkdudeCool! I was asking about s/conforming specifically which was a spelling error, it should be https://clojuredocs.org/clojure.spec.alpha/conformer#2020-08-1017:03borkdudeThis was the "transform while conforming" issue#2020-08-1308:41ikitommineed to publish a malli internal naming convention guide. my idea was that all public - starting functions are public for library extensions / advanced use, tracked separately in CHANGELOG from the public end user api. Using those for basic use cases is a sign of invalid usage. Those will break more than the user-facing public api, but so that it’s easy for advanced users adapt to changes, e.g. no silent breaking changes after initial public release.#2020-08-1114:35borkdude@ikitommi do the names :encode/conform and (mt/transformer {:name :conform}) match? why is mt/transform not called mt/encode?#2020-08-1114:36borkdudeah because it also has decode, gotcha#2020-08-1207:08ikitommithe transformation terms:
• transformation function, a function of A->B, e.g. converts strings to dates
• decoding, a process of transforming (invalid) values into potentially valid ones (IN*), m/decoder& m/decode*
• encoding, a process of transforming (valid) values into something else (OUT), m/encoder & m/encode
• transformer, a top-level component that maps Schemas with transformation functions (“json-transformer transforms strings to dates, but not strings to numbers”). Needed in encoding and decoding, mt/transformer
• named transformer, If a transformer has :name defined, Schemas can define their transformation functions (for both encoding & decoding) using Schema properties
• interceptor, a component that bundles transforming functions into transforming phases
• transforming phase, either :enter or :leave, timing when a transformation function is applied in the chain (before the fact or after the fact)
• interceptor chain, a sequence of interceptors that is used to run the (optimized chain of) transformation functions from interceptors in correct order
• transformation chain , transformers compose too: (mt/transformer {:name :before} mt/json-transfomer {:name :after})#2020-08-1210:33eskosCould be valuable to add these as glossary doc to the repository (`docs/glossary.md`) to document them 👍#2020-08-1207:08ikitommilot of things inside to make things composable and fast, the basic user doesn’t have to know much of the details.#2020-08-1207:23ikitommi(m/decode
[:map
[:x {:default 42} int?]
[:y [:vector {:decode/before #(str/split % #",")
:decode/after {:leave #(mapv inc %)}} int?]]]
{:y "1,2,3"}
(mt/transformer
{:name :before}
(mt/default-value-transformer)
(mt/string-transformer)
{:name :after}))
;{:y [2 3 4]
; :x 42}#2020-08-1207:24ikitommi(m/decode
[:map
[:x {:default 42} int?]
[:y [:vector {:decode/before #(str/split % #",")
:decode/after {:leave #(mapv inc %)}} int?]]]
{:y "1,2,3"}
(mt/transformer
{:name :before}
(mt/default-value-transformer)
(mt/string-transformer)
{:decoders {'int? #(* % 2)}}
{:name :after}))
;{:y [3 5 7]
; :x 84}#2020-08-1709:15Shuai Linis it possible for my transformer/decode to know the current path when decoding? e.g. in the below example, I'd like to replace the <location> with :k1
(ma/decode
[:map
[:k1 [:map {:decode/foo #(assoc % :location :<location?>)}
[:x int?]]]]
{:k1 {:x 1}}
(mt/transformer
{:name :foo}))
;; => {:k1 {:x 1, :location :<location?>}}#2020-08-1709:17Shuai Linthe background story is I want to generate a unique key for each such map based on its path in the whole tree#2020-08-1709:45ikitommi@linshuai2012 not at the moment, but you can do the following:
1) use m/walk to transform the schemas by adding a :in property to schemas (Schema :path and value :in are available for walkers)
2) create a decoder that uses the interceptor :compile hook to access the Schema at decoder creation time#2020-08-1709:46ikitommithere are few pending PRs around walkers, not 100% sure what is in the current master.#2020-08-1709:48Shuai Lin@ikitommi thx! I'll take a look at this approach. Currently I'm trying a solution that
1. leaves a place holder when decoding, and then
2. after decoding the whole tree, walk my tree on my own (and accumulate the path), and for each such map, use clojure.walk/prewalk to replace the placeholder with the current path#2020-08-1709:50ikitommipretty sure it's not very performant that way. There is an example of attaching a generated sample value to all schemas in the readme.#2020-08-1709:51ikitommiIt uses the m/schema-walker , you need the plain walker to access the :in data. But otherwise, the 1) should be copyable from that.#2020-08-1712:03Shuai LinJust realized it's not what I want, the schema walker walks the schema, not the actual data.
I still need to walk the actual data to get the pat to generate the unique key for each map in the tree based on its actual location. Also, my schema is recursive - think about a file system, where there are directories as maps, files as leaves, and sub-directories as child maps.#2020-08-1712:40ikitommiwalk + decode would work for non-recursive schemas. Please write an issue, I'll think about the solution.#2020-08-1715:26Shuai LinThx! I will.#2020-08-1709:52ikitommialso, the interceptor :compile hook could publish more information about the context to the transformers, last call to break the api before freezing things. Please write an issue if you want that#2020-08-1709:53ikitommiIt is now a callback fn with args of schema value, could be schema value path in root-schema for example#2020-08-1709:57Shuai Linthe m/walker apporach looks promising, I'll try it next#2020-08-1710:18CaseyI'm just getting started exploring malli#2020-08-1710:19Caseyhttps://github.com/metosin/malli#mallicorebase-schemas seem to indicate if I want a :int or :int-in schema, I'd have to register it myself, is that correct?#2020-08-1710:24ikitommi@ramblurr you can just create a function that returns a form for it:
(defn int-in [min max]
[:and int? [:fn '(fn [x] (< min x max))]])
#2020-08-1710:25ikitommibut you should add a :gen/gen for it too. I think there should be a built-in for that (and for dates too)#2020-08-1710:26ikitommi[:int {:min 1, :max 10}]
[:date {:min "2020-08-16", :max "2020-10-10"}]
kinda things#2020-08-1710:28ikitommithe raw impl would look much like :string: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L772-L805#2020-08-1710:28Caseyah great, i see#2020-08-1710:28Caseyis there an aggregated list of built ins somewhere (with docs as to their properties)#2020-08-1710:29Casey> I think there should be a built-in for that (and for dates too)
Do you mean you think this built-in already exists, or you had the idea that it should be added ? 🙂#2020-08-1710:29ikitommi(m/default-schemas) gives a list of all. the source code is currently the best source of descriptions.#2020-08-1710:29ikitommishould be added 🙂#2020-08-1710:30ikitommithe properties - will add description of those using malli (eat your own..), but not there yet.#2020-08-1710:33CaseyUsing the int-in function def you gave an example for above.. if you added it to a registry {:int-in (m/fn-schema :int-in int-in)} , how could it be consumed later? In fact, this wouldn't work right?#2020-08-1710:35CaseyYou could do something like {:int-in-0-10 (m/fn-schema :int-in-0-10 (partial int-in 0 10))} I suppose#2020-08-1710:44ikitommi(defn int-in [min max]
[:and int? [:fn `(fn [x] (< ~min x ~max))]])
(def int-in-1-10 (int-in 1 10))
(validate [:tuple int-in-1-10 (int-in 10 100)] [2 12])
; => true
(form [:tuple int-in-1-10 (int-in 10 100)])
;[:tuple
; [:and int? [:fn (clojure.core/fn [malli.core/x] (clojure.core/< 1 malli.core/x 10))]]
; [:and int? [:fn (clojure.core/fn [malli.core/x] (clojure.core/< 10 malli.core/x 100))]]]
#2020-08-1710:46ikitommiif you write an issue about the :int as built-in, happy to add that. much cleaner#2020-08-1711:15CaseyAm I correct that you can't use the generic :int-in as defined above in a registry, because the "registered" specs must be predicate predicate functions (single value as input)?#2020-08-1711:24ikitommi• registered schemas are one of: 1) IntoSchema instance (e.g. something that take the schema syntax and return a Schema. 2) Schema instance, 3) Schema syntax.
• m/fn-schema just takes a predicate fn, which is only used to build a validator for the schema
• there could be more helpers to build custom schemas easier, something between m/fn-schema (here: too simple) and writing IntoSchema impl (here: too much work) by hand, but currently, there is not.#2020-08-1711:26ikitommiI think I’ll extract the code from :string so that :date, :number, :date-time etc can reuse most of it (e.g. the :min + :max handling of via properties, effecting both validation and value generation)#2020-08-1711:28ikitommialso, you could build your own things with it easily: {:registry {:bigdec (m/-ranged-pred-schema {:pred bigdec?, :range-pred …, :range-gen ...})}}#2020-08-1712:03Shuai LinJust realized it's not what I want, the schema walker walks the schema, not the actual data.
I still need to walk the actual data to get the pat to generate the unique key for each map in the tree based on its actual location. Also, my schema is recursive - think about a file system, where there are directories as maps, files as leaves, and sub-directories as child maps.#2020-08-1714:35Casey@steveb8n I've been playing with your demo (https://github.com/stevebuik/fork-malli-ideas) , it's a really nice approach to form validation.#2020-08-1722:14steveb8n@UE35Y835W nice to see some interest#2020-08-1907:32LuYeah pretty cool!! 😎 #2020-08-1912:44Luin 2.1.4 you can add a keywordize-keys true option to work exclusivley with keywords/namespaced keywords instead of strings, also in the validation#2020-08-1714:36CaseyThe keyword <-> string transformations are clever too, though don't work when namespaced keys are used#2020-08-1715:40CaseyI'm working around this using
(defn unkeyword [m]
(map-keys #(subs (str %) 1) m))
at the end of validator-for-humans to turn :foo/attr-> "foo/attr" .. and of course using "foo/attr" as the name in the form form. Works well enough, if a little dirty.#2020-08-1714:37ikitommi@ramblurr https://github.com/metosin/malli/pull/243/commits/7d335e916d9769d4039b69e1475f04f3239ac5c7#2020-08-1714:38ikitommimerged in master#2020-08-1714:39Caseyfirst look comment: the max range should be exclusive, not inclusive. That's a pretty standard trope across all range checks in almost any language I know#2020-08-1714:39ikitommiwith m/-simple-schema it should be easy to add new schemas that use properties in validation:
(-simple-schema {:type :double, :pred double?, :property-pred (-min-max-pred identity)}))#2020-08-1714:40ikitommi:thinking_face:#2020-08-1714:40ikitommispec has that, test.check uses inclusive for both.#2020-08-1714:40ikitommicould you link some external wisdom for that?#2020-08-1714:40Caseyhttps://clojuredocs.org/clojure.spec.alpha/int-in-range_q is exclusive#2020-08-1714:41ikitommihttps://clojure.github.io/test.check/clojure.test.check.generators.html#var-large-integer* is inclusive#2020-08-1714:42ikitommiJSON Schema has inclusive: https://json-schema.org/understanding-json-schema/reference/numeric.html#2020-08-1714:43Caseyand so s/int-in and s/double-in are exclusive, https://clojuredocs.org/clojure.core/range is exclusive,#2020-08-1714:43Caseyjson schema supports inclusive and exclusive options 😛#2020-08-1714:46Caseyto be clear, i'm advocating only the max be exclusive, look at pretty much and langugage: python's list slice operator, java's IntStreams,#2020-08-1714:47Caseyhere's an argument why: https://wrschneider.github.io/2014/01/07/time-intervals-and-other-ranges-should.html (starting off about time, but then at the end mentions integer ranges)#2020-08-1714:48CaseyHaha, and https://stackoverflow.com/questions/8441749/representing-intervals-or-ranges is a link to https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html by Dijkstra on the topic#2020-08-1714:48ikitommithanks. will read those.#2020-08-1714:50CaseyThat said, I don't need to die on this hill 🙂 Just sharing my experience that generally when I see a range in an api, I assume (and assumed others did too!) that it was inclusive start and exclusive end. As long as it's documented, it'll be ok either way#2020-08-1717:25ikitommimeanwhile:
(mg/generate
[:map
[:string :string]
[:int :int]
[:double :double]
[:boolean :boolean]
[:keyword :keyword]
[:symbol :symbol]
[:qualified-keyword :qualified-keyword]
[:qualified-symbol :qualified-symbol]]
{:size 42, :seed 42})
;{:string "¦®GÏVá@£°5o,&µ7\rØã",
; :int -1251,
; :double -0.03125,
; :boolean true,
; :keyword :WD_VS_-r,
; :symbol k5K_2i,
; :qualified-keyword :M8qL/u?RAmf,
; :qualified-symbol x/y0T}
related: https://github.com/metosin/malli/issues/25#2020-08-1912:15CaseyGiven a registry map, is there a function that will return a list of the qualified keys in the registry?#2020-08-1914:55ikitommino, there isn't#2020-08-2006:07ikitommi... but just filter or reduce the keys?#2020-08-2007:00CaseyYup, that's what I did 🙂#2020-08-2007:03CaseyI'm finding malli very useful for client and server side entity validation. When you have an entity that a client can submit, but is only allowed to submit a subset of all possible attributes and the other attributes are populated by the server. In a ns for an entity, I have one main custom registry (which is just a map!) that defines all possible attributes of the entity, then I build little schemas that pick out the "views" of what a valid entity looks like at different stages in its lifecycle.#2020-08-2011:45CaseyWhat's the story behind the name malli, is there one?#2020-08-2012:28ikitommihttps://www.slideshare.net/metosin/malli-inside-datadriven-schemas#21 (slide 21)#2020-08-2113:07CaseyFollowing the local registry example at https://github.com/metosin/malli#local-registry calling (malli.core/schema? Adult) => false . This surprises me..#2020-08-2113:09CaseyIs it not a schema? Why not?#2020-08-2113:12ikitommiit’s just data?#2020-08-2113:23CaseyAh, I see. I see it expects a Schema protocol.... and you can use m/schema to turn the data into a Schema entity#2020-08-2113:24ikitommiI think the m/schema? is not that useful, could be removed.#2020-08-2113:25ikitommias most of the functions consume data, Schema or IntoSchema .#2020-08-2113:25Caseym/schema? is used in some fdef specs in gungnir#2020-08-2512:09Elso"No implementation of method: :-form of protocol: #'malli.core/Schema found for class: clojure.core$int_QMARK_"
Version bumped from "0.0.1-20200404.091302-14" to "0.0.1-SNAPSHOT".
Validating with
(->> (m/decode schema data mt/string-transformer)
(m/explain schema)
me/humanize))
and a schema like
[:map
[:something [:map
[:a int?]]]
and input data like
{:something {:a 10}}#2020-08-2512:09ElsoWhat's going on here?#2020-08-2513:19ElsoApparently, my example was not telling the important part and the breaking change was that this:
[:map
[:issue keyword?
:user-endpoint string?]]
used to work but is now required to be
[:map
[:issue keyword?]
[:user-endpoint string?]]#2020-08-2513:19ElsoWhich seems quite reasonable#2020-08-2515:56ikitommiplan is to describe malli schema syntax using malli to get humanized errors on invalid syntax#2020-08-2906:39ikitommi🎉#2020-08-3011:17borkdudeCongrats on the CT grant :)#2020-08-3011:17borkdudeWill malli have something like spec/fdef + instrument + unstrument? And will the sequential destructuring be part of the first production release?#2020-08-3015:51ikitommi@borkdude the first release is planned to be just the current and (hopefully) immutable core on which everything builds on, Here’s the roadmap for the together funding:
1) get a stable release out! lot's of small and some bigger design decisions, tracked via metosin/malli#116
2) help early adopters (users and libraries like reitit, regal, aave and gungnir) to upgrade to use the initial version
After the release, would work on the following:
3) finalize sequence schemas,
4) enhance developer tooling:
- function schemas with clj-kondo integration
- pull out and reuse the reitit development time error pretty printer as a separate library ()
5) implement pluggable schema inference
6) parsers
#2020-08-3015:53ikitommidestucturing might be easy to implement using the current explain api, need to visit that for the first release to see will it require changes to the m/explain format.#2020-08-3015:54ikitommiVincent has done a great initial work already with the sequence schemas, will dig into that soon.#2020-09-0119:03Vincent CantinThx ^_^
I recommend to take the impl from Minimallist as a reference, as it is the most up-to-date and (most importantly) well tested.
The work remaining to be done is integration and optimization.#2020-09-0209:35ikitommiwas just about to ask from which codebase should I look this for. Thanks!#2020-09-0210:48Vincent Cantinthe latest commit of the main branch all-work-and-no-play#2020-08-3015:54ikitommiso, no, but hopefully soon has both.#2020-08-3015:54borkdude:thumbsup:#2020-08-3015:59ikitommimerged https://github.com/metosin/malli/pull/249. Adds :double, `:boolean`, `:keyword`, `:symbol`, `:qualified-keyword`, `:qualified-symbol` and `:uuid`. There is now m/-simple-schema to easily build custom schemas using properties in validation. Could rewrite m/-predicate-schema to use that too.#2020-08-3106:08ikitommihttps://github.com/metosin/malli/pull/212 would remove m/map-entries in favor of m/entries retuning a sequence of clojure.lang.MapEntry. m/children is quaranteed to return the tupl-3 of [key properties schema] in the future. This is a breaking change, so comments welcome.#2020-08-3106:10ikitommirelates to 5 issues and 2 PRs, one of the last design decisions that needed re-visiting.#2020-09-0120:55shaunxcodeHave recursive schemas been implemented now? I know there was a ticket (which I cant find right now) where various versions were being discussed.#2020-09-0121:07jhacks@shaunxcode It looks like this might be the discussion: https://github.com/metosin/malli/pull/117 and maybe this is the implementation?: https://github.com/metosin/malli/pull/209 but I’m just getting familiar with malli#2020-09-0121:10shaunxcodethanks that is it exactly!#2020-09-0121:14jhacks@shaunxcode I just saw this section in the readme: https://github.com/metosin/malli#recursive-schemas which might be helpful too#2020-09-0121:39jhacksI was following along with this code from the README.md:
;; regexs work too
(mg/generate
[:re #"^[a-zA-Z0-9._%+-]
In order for it to work I needed to explicitly add test.chuck as a dependency, or else I would get this error:
Execution error (IllegalStateException) at malli.generator/eval20132$fn (generator.cljc:135).
Attempting to call unbound fn: #'malli.generator/-re-gen
If that’s the expected behavior (and I didn’t mess something up), maybe it would be good to add a note to the README.md under https://github.com/metosin/malli#value-generation about adding test.chuck as a dependency?#2020-09-0204:59ikitommi@jhacks documented the dependency to README and made it fail better: https://github.com/metosin/malli/commit/5c3689fd42f2c73253a0f81a88b23e20a3b6417b#2020-09-0205:02ikitommiI believe regal has regex generators for both clj & cljs, could use that instead. It’s still wip according to README:
> The following aspects have known issues or are otherwise untested or incomplete, and you can expect them to change significantly as we further develop them:
> * Creating test.check generators from regal forms#2020-09-0205:03ikitommimaybe @plexus could verify the readiness there? (there is also a schema type for malli in regal, which is awesome!)#2020-09-0214:09ElsoIs there something like nillable in malli?
Because
(malli.core/validate [:map [:a {:optional true} string?]] {:a nil})
=> false
(malli.core/validate [:map [:a {:optional true} string?]] {})
=> true
this is rather odd to me.#2020-09-0214:13jeroenvandijkI’m wondering if Malli supports lazy registries. I’m guessing it should be possible with the registry protocol. Anyone know of examples?#2020-09-0214:14jeroenvandijkI want to build an AWS Cloudformation validator, but I don’t want to load all schema files upfront. By doing it lazy I hope to win some startup time#2020-09-0214:31ikitommiSounds like a great idea. Just implement something lazy behind malli.registry/Registry and plug it in.#2020-09-0214:35jeroenvandijkThanks! I’ll give it a try#2020-09-0214:47jeroenvandijkSo the value defines what needs to be looked up. E.g. {:Type "AWS::EC2::Instance" …} would need to be validated with the schema for "AWS::EC2::Instance" , so I need to inspect the value before I know the type. Might make it a bit trickier#2020-09-0214:27jhacks> documented the dependency to README and made it fail better
Thanks, @ikitommi! 🙌#2020-09-0214:36ikitommi@d.eltzner012
(m/validate [:maybe :string] nil)
; => true
(m/validate [:maybe :string] "sheep")
; => true
#2020-09-0214:52ElsoThanks a lot!#2020-09-0215:36jhacks> I believe regal has regex generators for both clj & cljs, could use that instead
Wow, thanks for mentioning regal, I was not aware of it: https://github.com/lambdaisland/regal#use-with-malli#2020-09-0305:23ikitommi@jeroenvandijk wanted to test the lazy registries.
(require '[malli.core :as m])
(require '[malli.registry :as mr])
Given a data-source that can map names to schemas:
(def schema-provider
{"int" :int
"map" [:map [:x "int"]]
"maps" [:vector "map"]})
We can compose a registry that uses both local and lazy/external resolving:
(defn LazyRegistry [default-registry]
(let [cache* (atom {})
registry* (atom nil)]
(reset!
registry*
(mr/composite-registry
default-registry
(reify
mr/Registry
(-schema [_ name]
(or (@cache* name)
(do (println "loading" (pr-str name))
(when-let [schema (schema-provider name)]
(swap! cache* assoc name (m/schema schema {:registry @registry*}))
schema))))
(-schemas [_] @cache*))))))
(def registry (LazyRegistry m/default-registry))
Using the registry (either swap the m/default-registry or pass as argument:
(count (mr/-schemas registry))
; => 125
(m/validate "map" {:x 1} {:registry registry})
;loading "map"
;loading "int"
; => true
(m/validate "map" {:x 1} {:registry registry}) ;; cached
; => true
(count (mr/-schemas registry))
; => 127
(m/validate "maps" [{:x 1}] {:registry registry})
;loading "maps"
; => true
(count (mr/-schemas registry))
; => 128
Schemas are first class :refs:
(m/schema "map" {:registry registry})
; => "map"
(m/-deref (m/schema "map" {:registry registry}))
; => [:map [:x "int"]]
Hope this helps.#2020-09-0307:56jeroenvandijk@ikitommi Thanks for sharing. I think it’s almost what I need. I’m puzzling how to deal with the (lazy) dispatch on a map key. In clojure.spec I would use multimethods and multispec:
(defmulti resource-type :Type)
(s/def :aws.cfn/resource (s/multi-spec resource-type :Type))
;; Some random examples
(defmethod resource-type "AWS::AmazonMQ::Broker" [_] :aws.amazon-mq/broker)
(defmethod resource-type "AWS::AmazonMQ::Configuration" [_] :aws.amazon-mq/configuration)
(defmethod resource-type "AWS::ApiGateway::Account" [_] :aws.api-gateway/account)
(defmethod resource-type "AWS::ApiGateway::ApiKey" [_] :aws.api-gateway/api-key)
...
If I can do this dispatch somehow, with your suggestion I think I have all I need#2020-09-0308:02jeroenvandijkI’ll study the :multi schema and see if that is the missing piece#2020-09-0308:08ikitommis/multi-spec is open & mutable, :multi is closed & immutable.#2020-09-0308:09ikitommiso here, I think a lazy multi variant would be needed.#2020-09-0308:12ikitommia) lazy multi, with immutable values
[:multi {:dispatch :type, :children children-fn}]
b) mutable multi, backed by a custom (mutable) multimethod:
[:multi {:dispatch :type, :children my-multimethod}]#2020-09-0308:12ikitommi… actually would be the same code, it’s in user-space whether to allow overriding the keys.#2020-09-0308:13ikitommishould not be many loc to implement#2020-09-0308:29jeroenvandijkThanks. Makes sense. I’ll try to adapt https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L796#2020-09-0308:40ikitommiIf you make a PR, would like that the default case (e.g. no :children key set) will not slow down -> the entry parsing will happen at schema creation time. for the case of dynamic childs - it would happen at runtime.#2020-09-0308:41ikitommione question is: what happens if you create a validator, explainer or generator out of that schema: should the current children be used or should those be dynamic too.#2020-09-0308:42ikitommie.g. if you add a branch after creating a validator, will the validators before that see it or not.#2020-09-0308:45jeroenvandijkWith clojure.spec I have one spec that contains all types. This gives you a suggestion in case the dispatch on type fails. E.g.
(s/def :cfn.all/Type #{"AWS::AmazonMQ::Broker" "AWS::AmazonMQ::Configuration" "AWS::ApiGateway::Account" "AWS::ApiGateway::ApiKey" "AWS::ApiGateway::Authorizer" "AWS::ApiGateway::BasePathMapping" "AWS::ApiGateway::ClientCertificate" .....})
This is not ideal either because it doesn’t have spell-check functionality. But to answer your question, I don’t think, at least for my use case, everything has to be dynamic#2020-09-0314:39jeroenvandijk@ikitommi The start of this seems to be simple indeed https://gist.github.com/jeroenvandijk/59d22a726cda2158c01b9d63790aec50#file-malli_lazy-clj-L80 I’ve only added the validator part, not sure if the transformers and explainers will make things more painful#2020-09-0315:57ikitommi@jeroenvandijk just to Make sure: you do know all the possible dispatch keys in advance?#2020-09-0315:57ikitommi(if so, there might be a simpler solution)#2020-09-0316:17jeroenvandijkYeah all the dispatch types are known in this case. The raw schema data is close to 1mb. So that's the main reason to do it lazy#2020-09-0319:12ikitommi@jeroenvandijk This would be a small change in :ref impl:
(defn LazyRegistry [default-registry f]
(let [cache* (atom {})
registry* (atom nil)]
(reset!
registry*
(mr/composite-registry
default-registry
(reify
mr/Registry
(-schema [_ name]
(or (@cache* name)
(do (println "loading" (pr-str name))
(when-let [schema (f name)]
(swap! cache* assoc name (m/schema schema {:registry @registry*}))
schema))))
(-schemas [_] @cache*))))))
(def registry
(LazyRegistry
m/default-registry
{"map1" [:map [:type [:= "map1"]] [:x :int]]
"map2" [:map [:type [:= "map2"]] [:y :int]]
"map3" [:map [:type [:= "map3"]] [:z :int]]}))
(m/validate
[:multi {:dispatch :type}
["map1" [:ref "map1"]]
["map2" [:ref "map2"]]
["map3" [:ref "map3"]]]
{:type "map3", :z 1}
{:registry registry
::m/lazy-refs true})
;loading "map3"
;=> true#2020-09-0319:14ikitomminew option :malli.core/lazy-refs that would control if the :refs are checked eagerly or lazily#2020-09-0319:15ikitommior there could be a :lazy variant of :ref to make things explicit.#2020-09-0319:16ikitommior a new property :lazy to :ref to mark it being lazy:
[:ref "map1"]
[:ref {:lazy true} "map1"]#2020-09-0319:16ikitommiI think that’s actually good.#2020-09-0319:23ikitommihttps://github.com/metosin/malli/pull/252#2020-09-0320:05ikitommiactually, we can push all the changes from user api (e.f. schema props) into extender api (here: lazy registry impl). This allows to write fully lazy multis:
[:multi {:dispatch :type}
"AWS::AmazonMQ::Broker"
"AWS::AmazonMQ::Configuration"
"AWS::ApiGateway::Account"
"AWS::ApiGateway::ApiKey"
"AWS::ApiGateway::Authorizer"]#2020-09-0320:07ikitommi(`:multi` uses the entry-syntax, like :map which allows single-value elements if they are valid schema reference types, now: just qualified keywords, should be strings too)#2020-09-0408:46ikitommi#2020-09-0408:47ikitommithat look ok @jeroenvandijk?#2020-09-0408:48jeroenvandijk@ikitommi it looks perfect! I need to wrap my head around it, but the demo looks exactly what I want. I’ll try to convert a spec project today and give some feedback#2020-09-0408:55ikitommigreat! changes in the core are in this commit: https://github.com/metosin/malli/pull/252/commits/3a5c7f35141e32eea3b374b6a81e706377a44d26, need to add polish the code before merging in, most likely this week.#2020-09-0416:38jeroenvandijkDoes Malli have an equivalent of (clojure.spec.alpha/map-of string? string) .e.g {"any-string-key" "any-string-value"} ?#2020-09-0416:51jeroenvandijkAs an alternative this seems to work:
(require '[malli.core :as m])
(defn map-of [k v]
(clojure.walk/postwalk-replace
{::k k
::v v}
'[:fn (fn [x] (and (map? x)
(every? (fn [[k v]]
(::k k)
(::v v))
x)))]))
(m/validate (map-of 'string? 'string?) {:a 1})
#2020-09-0417:03ikitommi[:map-of string? string?]#2020-09-0417:04jeroenvandijkah 🙂#2020-09-0417:04jeroenvandijkThanks#2020-09-0417:15jeroenvandijkWould it make sense to add validation on the schema definition? e.g. i did this in a nested schema
(m/validate '[:map string? string?] {})
I had no idea of course and I was looking for the place that was causing this message
java.lang.UnsupportedOperationException: count not supported on this type: Symbol
#2020-09-0418:04ikitommiIntoSchema protocol will have a methods that describe both children and properties as malli schemas. After that, malli validates the malli schemas :) That requires the regex-schemas, which is the next thing. So, yes, but not yet.#2020-09-0418:06ikitomminot sure if there is an issue of that, should be.#2020-09-0417:39jeroenvandijkI was thinking of trying the non-lazy approach first, but I think the lazy version will be easier to debug 😅 I can print what faulty schema is being loaded just before it crashes#2020-09-0418:23jeroenvandijk@ikitommi So far the lazyness works really nice. I’ll let you know when I’m done#2020-09-0418:41jeroenvandijkIs it possible to somehow add the spellcheck on the dispatch as well? Would be super fancy 🙂 E.g. when I type
"AWS::AppSync::ApiK" ==> you misspelled "AWS::AppSync::ApiKey"
#2020-09-0418:43jeroenvandijkI’ve added a try/catch around the schema loading. Makes it much easier to debug a wrong definition:
(fn [type registry]
(let [definition (lookup-fn type)
schema (when definition
(try
(m/schema definition {:registry registry})
(catch Exception e
(throw (ex-info (str "Error while loading " type ": " (pr-str (ex-message e))) {:definition definition})))
))]
(println "loaded" (pr-str type))
schema))#2020-09-0419:22ikitommishould be possible. The spell checking reads the explain output. Just need to ensure :multi emits error on invalid dispatch key that the spell checker understands. Internally, :map and :multi both use the entry-syntax, so might be just few loc.#2020-09-0419:39jeroenvandijkAs far as I can see the spelling feature is focussed on the misspelling of keys and not of values. I probably missing something though. I was missing this spelling on values also when I was spec with expound and spell-check. Maybe it’s a different kind of problem#2020-09-0419:47jeroenvandijkah wait i think i see some pointers#2020-09-0419:24ikitomminext step would be to make clj-kondo use malli somehow, so we would get static inspection.#2020-09-0419:25ikitommiand the next would be to have auto-complete via lsp.#2020-09-0502:43rutledgepaulvI was wondering if lsp integration was in the cards 🙂 I would love to be able to write a malli schema and from it receive strong editor support for data that needs to conform to that schema (like some tools provide for xsd).#2020-09-0419:26ikitommi(no idea how to do the last, clj-kondo should be fun to try)#2020-09-0419:36jeroenvandijkSounds awesome!#2020-09-0420:01borkdude@ikitommi is this related to what you showed on ClojureD? I'm not sure what you mean by clj-kondo using malli#2020-09-0507:48ikitommiThe demo on ClojureD was schematized malli fn's emitting clj-kondo type definitions. I'll make a real impl out of that now with the clj-together funding. Not the first thing, but will most likely need help with that @borkdude, will poke you when about to do that.
The second thing, which would be a nice experiment is if malli itself could be used inside clj-kondo to validate both malli data and schemas, e.g.
(do-it {:Type "AWS::AppSync::ApiKey"
:Descriptionz "kikka"})
.. would emit clj-kondo errors:
{:ApiId ["missing required key"]
:Descriptionz ["should be spelled :Description"]}#2020-09-0507:51ikitommiautocomplete-stuff: I don't know how to do those, and most likely don't have to study that, but happy to help on Malli side if someone want's to try that out#2020-09-0508:27ikitommiwith the :multi dispatch key validation, it would yield:
(do-it {:Type "AWS::ApiGatway::UsagePlan"
:Description "kikka"
:UsagePlanName "kukka"}
.. would emit clj-kondo errors:
{:Type ["should be spelled \"AWS::ApiGateway::UsagePlan\""]}#2020-09-0508:29borkdude@ikitommi I tried a similar thing using the type system in clj-kondo. It can be done, but it only works at places where literal maps are passed. So when you use assoc et al, it already breaks (although I think I have some logic which tries to account for that, it's been a while)#2020-09-0508:29borkdudeas an experiment, it'd be interesting what comes out of it#2020-09-0508:32borkdudeI recently listened to a podcast with Tony Kay. He is working on something similar called GuardRails Pro which will be closed source#2020-09-0508:33borkdudeIt would be great to have a free alternative#2020-09-0513:13ikitommiLooked up and listened the ClojureScript Podcast about GRP, sounds interesting.#2020-09-0513:14ikitommiInteresting times ahead, sounds like Clojure is getting mature :)#2020-09-0513:33ikitommi@jeroenvandijk small changes:
• :dispatch is mandatory in :multi, e.g. can’t set via Schema creations opts => this makes it visible for error handling
• malli.error/with-spell-checking now understands :multi dispatch values if the :dispatch value is a keyword#2020-09-0513:33ikitommihttps://github.com/metosin/malli/pull/252/commits/fecd792d02e20a9a22730ea0163b661c539eaa79#2020-09-0513:33ikitommi(-> (m/explain
Schema
{:Type "AWS::AppSync::ApiKey"
:Descriptionz "kikka"})
(me/with-spell-checking)
(me/humanize))
; loaded "AWS::AppSync::ApiKey"
; => {:ApiId ["missing required key"]
; :Descriptionz ["should be spelled :Description"]}
(-> Schema
(m/explain
{:Type "AWS::ApiGatway::UsagePlan"
:Description "kikka"
:UsagePlanName "kukka"})
(me/with-spell-checking)
(me/humanize))
; => {:Type ["did you mean AWS::ApiGateway::UsagePlan"]}#2020-09-0513:35ikitommifull gist here: https://gist.github.com/ikitommi/06143540e6259323b95253e87e1ef710#2020-09-0513:40jeroenvandijkWow, super nice. Thanks!#2020-09-0706:33ikitommifirst spike of heterogenous sequences, aka “regex schemas”. Looked into seqexp, clojure.spec and minimallist for inspiration. Slicing a [1 2 3 "4" "5" 6 7 8 9 "10"] sequence with spec-equivalient:
(s/cat :1 (s/+ int?)
:2 (s/+ string?)
:3 (s/+ int?)
:4 (s/+ string?))
in malli, it would be:
[:catn
[:1 [:+ int?]]
[:2 [:+ string?]]
[:3 [:+ int?]]
[:4 [:+ string?]]]
or anonymus (uses vector indexes as keys)
[:cat
[:+ int?]
[:+ string?]
[:+ int?]
[:+ string?]]
some silly perf numbers with 10000 & time (and intermediate results):
Malli:
{:1 [1 2 3], :2 ["4" "5"], :3 [6 7 8 9], :4 ["10"]}
"Elapsed time: 35.535477 msecs"
Minimallist:
{:1 [1 2 3], :2 ["4" "5"], :3 [6 7 8 9], :4 ["10"]}
"Elapsed time: 386.464772 msecs"
Seqexp:
{:rest (), :match (1 2 3 "4" "5" 6 7 8 9 "10"), :1 (1 2 3), :2 ("4" "5"), :3 (6 7 8 9), :4 ("10")}
"Elapsed time: 686.08428 msecs"
Clojure.Spec:
{:1 [1 2 3], :2 ["4" "5"], :3 [6 7 8 9], :4 ["10"]}
"Elapsed time: 879.329458 msecs"#2020-09-0905:38Vincent Cantin@ikitommi You made my day ! I never tried any benchmark on my lib, I did not know it was faster than Clojure Spec 😃#2020-09-1018:06ikitommiawesome#2020-09-0709:36borkdudecool :)#2020-09-0710:05borkdude@ikitommi I think this new clj-kondo feature can help with annotations generated by malli in some other config file:
https://github.com/borkdude/clj-kondo/issues/992
Users can e.g. use {:config-paths ["malli-types"]}. If malli then spits out the type information in .clj-kondo/malli-types/config.edn, then it will work#2020-09-0710:05borkdudeThis is for the ClojureD-demoed functionality#2020-09-0710:06borkdudeI'm also considering a similar tool for clojure spec which inspects specs at runtime and then spits out a file. Maybe there could also be a CIDER middleware which populates things there as you eval code.#2020-09-0710:33borkdudebut if malli is going to have a defn like macro, I think making built-in functionality for it would also be cool. can we do this without depending on malli itself for keeping the dependencies of clj-kondo lean? e.g. it also doesn't depend on schema, it only recognizes the syntax#2020-09-0712:47ikitommi@borkdude good stuff. yes, the planned m/defn can emit the clj-kondo format directly, it’s a great start. Will keep an eye on your spec-tooling work too.#2020-09-0720:43tekacsReally looking forward to the m/defn support!
I forked aave to add ClojureScript support and m/humanize-d errors recently and am adding in a few additional things — I’ll try to comment in the m/defn threads once they’re live to see whether any of the features added can make it across when the time comes.
In particular, I’m binding up parameter and return types together (rather than separately), so that :multi / :or specs can be used as a poor-man’s (although runtime-dispatched) facsimile of overloading.
i.e.
[[[:vector int?] => [:map-of int? string?]]
[[:vector string?] => [:map-of string? int?]]]
translates to
[:or [:tuple [:vector int?] [:map-of int? string?]]
[:tuple [:vector string?] [:map-of string? int?]]]#2020-09-0720:43tekacscc/ @borkdude#2020-09-0720:45tekacsI’m also excited to use [:multi {:dispatch …}] instead of [:or] (the above are or-ed together), to see whether dispatch to a particular signature can be more sophisticated than simply ‘first matching signature’#2020-09-0806:11ikitommisounds great @tekacs, hopefully planning to push changes back to original Aave? Looking forward to hearing how the malli->kondo works with that. The. m/defn demo was a fork of Plumatic Schema s/defn, need to clean it up before use and open for comments. If the aave-syntax is already good, one option is just to add a fdef -malli decorator for existing functions into the core lib. Not sure. I believe Rich has been deep in the hammock thinking about new syntax for spec2 embedded defn syntax. Might take a while, but if becomes de-facto, malli could do that too.#2020-09-0806:14ikitommiPlumatic has btw s/conditional which might be handy with malli too. Like :multi but the keys are schemas to match the original value, e.g.
[:conditional
[vector? [:vector int?]]
[map? [:map [:x int?] [:y int?]]]
#2020-09-0816:14tekacsoh that’s interesting, yes (to both :conditional and :else as really helpful things to have)#2020-09-0914:15jhacksFollowing along with the example here: https://github.com/metosin/malli#custom-registry
(def registry
(merge
(m/class-schemas)
(m/comparator-schemas)
(m/base-schemas)
{:int (m/fn-schema :int int?)
:bool (m/fn-schema :bool boolean?)}))
(m/validate [:or :int :bool] 'kikka {:registry registry})
; => false
(m/validate [:or :int :bool] 123 {:registry registry})
; => true
It looks like m/fn-schema no longer exists. I tried using m/-fn-schema (which does exist) but that didn’t work.
How does this example work with the current code?#2020-09-1015:28ikitommifixed the README#2020-09-0916:04ikitommi@jhacks oh, the README is out of sync. This should work:
(m/-simple-schema {:type :boolean, :pred boolean?})#2020-09-0916:04ikitommithere seems to be (m/-predicate-schema :boolean boolean?) too#2020-09-0917:38jhacks@ikitommi Thanks, both methods work! Is there a way to register a default error message for schemas in the custom registry? For example, the custom :boolean schema has "unknown error" for the error message. Could I define a different message?#2020-09-1015:16ikitommi@jhacks currently no, but should be. wrote an issue about it: https://github.com/metosin/malli/issues/254 & will fix that soon.#2020-09-1015:19ikitommirenaming (& moving) the malli.error/ErrorSchema as malli.core/Humanized would make humanized errors more first class and would enable setting those easily when creating custom schemas - either via helpers of by reifying the protocols. Closure DCE will remove those anyway if not used in an app.#2020-09-1015:22ikitommicould also move the Generator protocol (not impls!) into core. Currently - if one want’s to implement fully custom Schma impl (like regal does), the lib must include all the support namespaces to access the Protocols. This drags is a lot of code that the end user might not need, making initial load time slower on clj and making the bundle size bigger on cljs.#2020-09-1017:50ikitommilooks legit.#2020-09-1018:00ikitommibig thanks to @vincent.cantin for the initial work on the issue! it’s now a full rewrite (protocols & vectors), but took the internal syntax from minimallist. minimallist inspired by malli, inspired by minimallist 😉#2020-09-1018:06ikitommiquite happy with the internal design. both validation and parsing use the same code, but when validating, the parsing results are not realized. that given example is 40µs on spec (both validation & parsing), malli is now: 6µs when parsing and 3µs on validation. not optimized yet, I think it could be made much faster still.#2020-09-1018:15ikitommi123 loc 🙂#2020-09-1023:48Vincent Cantinmicromallist#2020-09-1019:14borkdudelooks awesome#2020-09-1021:39tekacsI was considering making an issue to ask, but trying here first — is there a particular way you’d recommend doing a map on (or otherwise delegating to) existing schemas, @ikitommi?
Some examples are:
• Validating JS objects with a spec for their keys/values by using some of the implementation of [:map-of …] on a (decently efficient) cljs-bean.core/bean
• ^ similarly with JS arrays
•
Ideally I’d like to pass the underlying explainer’s output through and be able to humanize the aggregate schema, which I’m especially having trouble with.
So far I’ve done things like:
(defn transform-spec [transform inner-spec]
(let [explainer (m/explainer inner-spec)]
[:fn {:error/fn (fn [{:keys [value]} _] (explainer value))}
#(m/validate inner-spec (transform %))]))
; and
(defn js-obj [& map-spec]
(let [inner-spec (into [:map] map-spec)
explainer (m/explainer inner-spec)]
[:fn {:error/fn (fn [{:keys [value]}] (explainer value))}
#(->> (bean %) (m/validate inner-spec))]))
… but I’m breaking the explainer output no matter how I slice it.
The best approach for now I imagine is to directly implement the protocol(s) and do a bunch of delegation to the underlying schemas, but wondering if combinators/transformers are possible?#2020-09-1022:52jhacks@ikitommi Thanks for the very informative response. https://github.com/metosin/malli/issues/254 looks great! Also, sequence/regex schemas! Wohoo!#2020-09-1023:44Vincent Cantin@ikitommi now, you can finally use Malli itself to validate Malli’s models.#2020-09-1109:50ikitommi@tekacs not 100% sure about your use case, what about using transformers?#2020-09-1523:38tekacsI was hoping rather to do something more like gen/fmap but for malli types.
The hope being to be able to first run a transform on a value before validating it using an existing/builtin validator.
I can do this today using simple-schema or similar, but passing errors along is just tricky.#2020-09-2015:12ikitommidoes the new -simple-schema solve this?#2020-09-1109:55ikitommiI recently tested the end-to-end performance of JSON and realized we could plug malli into the jsonista pipeline: define a malli schema, create a jsonista functional-decoder out of it which picks just the defined keys from the JSON stream and runs (already optimized) value transformation for it. As there are no intermediate conversions, it should be MUCH faster.#2020-09-1110:01ikitommi• old-style json: stream --json-decode--> edn --schema-decode--> domain-data --json-encode--> string --ring-adapter--> bytes
• malli+jsonista: stream --json-and-schema-decode--> domain-data --json-and-schema-encode--> bytes#2020-09-1110:33borkdude@ikitommi That's exactly an issue that I posted a couple of years ago in the compojure-api channel :) Awesome that this will now be supported via malli!#2020-09-1110:34borkdudeThis is typically what you do in typed languages: write (de)serialization code which results in very efficient JSON processing. But in Clojure we typically just parse the entire blob#2020-09-1110:35borkdudeIf you have a working example of this, I'd be very interested#2020-09-1110:41borkdudesince jsonista is also based on jackson-core I might be able to switch/add from cheshire to jsonista without adding very much binary size. if we then also add malli and reitit and http-kit server ... in babashka I mean. small web-apps :)#2020-09-1114:55ikitommithat would be great! Would like to get vertx working with clj + graalvm at some point. After that, one could write a code that can run really fast with jvm, with low resources with graalvm and with bb for scripting.#2020-09-1114:57ikitommireitit+jsonista+porsas+pohjavirta is still one of the fastest jvm stacks in techempower db-query tests. It's a silly benchmark, but, still.#2020-09-1115:00ikitommihttps://www.techempower.com/benchmarks/#section=data-r19&hw=ph&test=db&a=2&f=35s1-1ekg-27xgcg-75-0-pfk-8vn0dc-kb4lg-13ydj4-8-0#2020-09-1115:01ikitommidon't have an example of the json+malli, just know it's possible. Will do.#2020-09-1523:38tekacsI was hoping rather to do something more like gen/fmap but for malli types.
The hope being to be able to first run a transform on a value before validating it using an existing/builtin validator.
I can do this today using simple-schema or similar, but passing errors along is just tricky.#2020-09-1315:49borkdudeI just realized that borkdude/dynaload isn't GraalVM friendly in that it uses require at runtime.
I made an issue for this: https://github.com/borkdude/dynaload/issues/6
I think we'll have to adapt to the CLJS model of dynaload, i.e. the require is on the user, dynaload only prints a warning#2020-09-1316:13borkdudeNote: it did work, but it's just not good for binary size and compilation time#2020-09-1319:49borkdudeDo you test GraalVM builds on CI?#2020-09-1709:53StefanHi all, somebody pointed me to Malli (thx @borkdude @hobosarefriends) because of the situation with spec being alpha and spec2 on the horizon. But then I noticed in the Malli readme that it says: “Pre-alpha”, which scares me a bit. Can somebody clarify the situation and whether or not this is a good time to start using Malli? Our situation by the way is that we have a big existing codebase without any sort of spec/schema, so in that respect we’re starting from scratch.#2020-09-1711:11jeroenvandijkAll the solutions are officially still alpha. Trying any of them is a good learning experience. If you don’t have specific requirements, I would suggest to start with clojure.spec (1) as it has the most examples and documentation. I think Malli is modelled like spec(2) so any learnings with clojure.spec will be applicable later in malli#2020-09-1711:15borkdudeAlthough @ikitommi mentioned that the goal of the current Clojurists Together funding is to make a stable (non-alpha) release?#2020-09-1711:16borkdudeWith spec it remains to be seen. It's been in alpha for over 4 years.#2020-09-1711:28jeroenvandijkGood point 🙂#2020-09-1720:28rschmuklerThere's some discussion on github that might be interesting: https://github.com/metosin/malli/issues/207#2020-09-1711:02jeroenvandijk@ikitommi A small heads up I have generated a Malli spec that can validate all of our cfn templates (around 30, pretty big and complete). I’ll try to publish it soon and point to some pitfalls. One thing that is hard to debug are (deeply nested) recursive schemas. Without the use of [:ref …] it will give a stackoverflow error without a good pointer. And I had to do a trick in the :multi dispatch to prevent :invalid-type errors and to stay compatible with spell-check. All in all it works pretty well! Thanks#2020-09-1714:53jeroenvandijk@ikitommi Here is the current state of my AWS cloudformation malli code https://github.com/jeroenvandijk/aws.cloudformation.malli/blob/master/src/adgoji/aws/cloudformation/malli/validation.clj#L15-L232#2020-09-1714:54jeroenvandijk(now public)#2020-09-1714:56jeroenvandijkMaybe useful information; I’ve reduced the usage of :or and used :multi instead to prevent super long lists of errors (reduces branching factor)#2020-09-1714:57jeroenvandijkI think I broke proper spell check feedback by not using a keyword as dispatch function. I’ll look into this soon again#2020-09-1711:17ikitommi@stefan.van.den.oord Just checked that last bullet on “last things before initial stable release” yesterday. will clean up corners and will ship the alpha, most likely next week.#2020-09-1711:18ikitommithe public api has been mostly stable since june, there has been small changes in the advanced user / extender api, and most likely will be after the first release.#2020-09-1711:19borkdude@ikitommi (Note my remarks on borkdude/dynaload. I'm currently solving a problem with GraalVM. Using resolve at runtime bloats the binary with +20MB. But it does work.)#2020-09-1711:19ikitommi1.0.0 might be soon too, after feedback from the community.#2020-09-1711:20ikitommi@borkdude sorry, read you message, but didn’t have time to answer. There is no GraalVM tests atm, should be.#2020-09-1711:20borkdudeI will publish dynaload 0.2.0 or so that will have breaking changes to fix this problem, but I'll notify you#2020-09-1711:20borkdudeand possibly make a PR#2020-09-1711:21ikitommithat would be great! GraalVM support is top priority, don’t want bloated binaries 🙂#2020-09-1711:22ikitommiwhen writing libraries, the goals matter a lot. few years ago, didn’t know much about perf. can’t add that later. now, the cljs bundle size, learned how to handle that. Next: GraalVM as a target.#2020-09-1711:22borkdudeyeah. I have a dynaload-graal-friendly branch, with script/graal-test.
currently this code still bloats:
#?(:clj (defn resolve* [sym]
;; TODO: this adds + 20MB to the GraalVM binary
#_(let [ns (symbol (namespace sym))
v (symbol (name sym))]
(when-let [ns (find-ns ns)]
(.getMapping ^clojure.lang.Namespace ns v)))))
so it's either in find-ns or in the namespace interop#2020-09-1711:27borkdudeyeah, it's the namespace interop#2020-09-1711:28borkdudeI think using this interop will make GraalVM think it should hold on to more that it actually needs#2020-09-1711:30borkdudeso what we can do is only resolve at compile time. The user should take care of requiring the lib namespace before they load the dynaloaded code, else it will be considered not there.#2020-09-1711:32ikitommiis there a :preload thing with deps? other than the -e from command line?#2020-09-1711:33borkdudeno#2020-09-1711:33ikitommihave this on my project Justfile:
# start backend nrepl
@backend:
clj -A:dev:test:common:backend -e "(require '[hashp.core])" -m nrepl.cmdline -i -C#2020-09-1711:34borkdudefor graalvm builds with deps.edn that may work#2020-09-1711:35borkdudewith the lein + uberjar approach it may not#2020-09-1711:35borkdudejust take care in your main to require those libs first, then it will be solved. we can make this behavior graalvm only for example by reading an environment variable or java property#2020-09-1711:37ikitommiwhere is borkdude/am-i-in-graalvm library?#2020-09-1711:38ikitommiso:
1. cljs: preload or direct require
2. jvm: just in a classpath
3. graalvm: direct require
, right?#2020-09-1711:38ikitommior:
2. jvm: just direct require#2020-09-1711:40ikitommican one differentiate if the lib is required or just in the classpath in 2? could be an option in dynaload?#2020-09-1711:41borkdude@ikitommi 1) I think in CLJS the order of require doesn't matter, because the check happens at runtime at the first deref. 2) this works because CLJ has runtime require 3) Like 1, but now the order matters, since we check at compile time, before the deref.
About checking the classpath: not sure how this would look. That's kind of the same as compile time resolve.#2020-09-1711:44borkdudeChecking whether you are in a GraalVM binary already works, but that's too late. You have to check at (Clojure) compile time.#2020-09-1711:49borkdudeSo based on setting -J-Dborkdude.dynaload.target=graalvm-native we could alter the behavior or 2 to 3#2020-09-1711:50borkdudeor -J-Dborkdude.dynaload.resolve-time=compile#2020-09-1713:35eskosre: graalvm library, shouldn’t that be called antioch ?)#2020-09-1721:31borkdude@ikitommi Pushed dynaload 0.2.1 with better support for GraalVM binaries:
https://github.com/borkdude/dynaload#graalvm#2020-09-1905:35ikitommiMalli now supports type-level properties (a breaking for extenders as there is a new protocol method in m/Schema) . Added a guide how to use them & also how to use non-registered Schemas.#2020-09-1905:35ikitommihttps://github.com/metosin/malli#custom-schema-types#2020-09-1905:39ikitommiIn short:
(def Over6
(m/-simple-schema
{:type :user/over6
:pred #(and (int? %) (> % 6))
:type-properties {:error/message "shuould be over 6"
:json-schema/type "integer"}}))
(m/into-schema? Over6)
; => true
(-> (m/explain Over6 5) (me/humanize))
; => ["shuould be over 6"]
(json-schema/transform [Over6 {:json-schema/example 7}])
; => {:type "integer", :example 7}#2020-09-1912:45pithyless(m/explain Over6 "not-an-int") would not return the most obvious error. What is the idiomatic way to reuse and compose custom types? Something like this?
(def Over6
[:and
pos-int?
(m/-simple-schema
{:type :user/over6 ;; is this required?
:pred #(> % 6)
:type-properties {:error/message "should be over 6"}}])#2020-09-1912:54pithylessI am currently using a custom version of -simple-schema (that allows setting custom error messages and generators). It's nice to see :type-properties added; maybe I can get rid of my custom code now. But I'm still struggling a bit with understanding how to best write custom types that don't need to re-implement predicate logic twice (once for validation, once for humanized errors); and how to compose things better.#2020-09-1912:56pithylessThis Over6 is a great example of a custom type, where ideally I could take advantage of all of the logic from e.g. pos-int? or [:int {:min 7}] and possibly add only additional :pred and :error/message#2020-09-1912:58pithylessOr I could just be wrong. But this is currently the one thing I don't understand on how to do well, in an otherwise fantastic library. :)#2020-09-1911:22schmeehello 👋 just curious if anyone has tried to derive Postgres schemas from Malli schemas (or Malli -> DB schemas in general)?#2020-09-1919:42dcjHere is what I did, This was my first Mailli use, not saying it is awesome or prefect:
Added properties to map schema that provide info about how to defiine a Postgres table to store the map:
(def Datafile
(m/schema
[:map {:closed true
:postgres/type :table
:postgres/schema "slm"
:postgres/table "datafile"
:postgres/key-encoder (comp csk/->snake_case_keyword remove-trailing-? name)}
[:id {:optional true
:postgres/type :column
:postgres/datatype :bigserial
:postgres/key :primary} int?]
[:serial-number {:postgres/type :column
:postgres/datatype [:varchar 64]
:postgres/null? false} string?]
[:filename {:postgres/type :column
:postgres/datatype [:varchar 64]
:postgres/null? false} string?]
[:extension {:postgres/type :column
:postgres/datatype [:varchar 8]
:postgres/null? false} string?]
[:create-time {:postgres/type :column
:postgres/datatype :timestamptz
:postgres/null? false} :zoned-date-time]
[:ingest-time {:optional true
:postgres/type :column
:postgres/datatype :timestamptz
:postgres/null? false} :zoned-date-time]
[:complete? {:optional true
:postgres/type :column
:postgres/datatype :boolean
:postgres/null? false} boolean?]
]
{:registry registry}))
Then I wrote functions that create a DDL text file from this info.
I also considered creating the table by connecting to the DB and directly creating it, but left that as an exercise for another day....
I also created functions that create a postgres-column-name-transformer
And I created functions edn->postgres and postgres->edn that take a schema and a map, and returns a map that has been transformed per the schema.
I found this pretty useful, and plan to continue to improve and use this going forward.
There are cases where there is also some related web API, that has its own keys/values, and I did something similar for that, e.g. added properties that define the keys and datatypes of the API, and am able to create api->edn and edn->api functions to transform back and forth....#2020-09-1919:53schmeecool, thanks for sharing! 😄#2020-09-2009:48ikitommi@pithyless good point about composition of things. Not sure myself what is a best practise with everything. Thinking aloud the ways to do things and when they should (or not) be used:
Advanced usage:
1. implementing IntoSchema is the last effort in doing things, e.g. adding a new :regal/regex type, which has custom explain, transform etc.
2. using -simple-schema is helper for 1 (but only for leaf-schemas).
Basic usage:
using schemas as data should be the common way of doing & composing things. One can set transformation rules, humanized error messages, json schema mappings etc. as schema properties. Two options for composing things:
1. Vars. just say (def Over6 (m/schema [:int {:min 6, :description "should be over 6}])) and use the Var. Caveat: inlines the forms: (m/form [:and int? Over6]) ; => [:and int? [:int {:min 6, :description "should be over 6"}]], making large schema hard to read
2. Via registry:, e.g. {"Over6" [:int {:min 6}]} , keeps the reference visible: (m/form [:and int? "Over6"]) ; => [:and int? "Over6"] keeping the schema forms clean too#2020-09-2009:56ikitommiadded support for m/type-properties -based transformations too. And a way for -m/simple-schema to create the Schema Instances based on actual Schema instance properties.#2020-09-2009:59ikitommiNot sure how useful this is, but as the malli lifecycle already supports this, just made it easy to use:
(testing "with instance-based type-properties"
(let [Over (m/-simple-schema
(fn [{:keys [value]} _]
(assert (int? value))
{:type :user/over
:pred #(and (int? %) (> % value))
:type-properties {:error/message (str "should be over " value)
:decode/string mt/-string->long
:json-schema/type "integer"
:json-schema/format "int64"
:json-schema/minimum value}}))]
(testing "over6"
(let [schema [Over {:value 6}]]
(testing "form"
(is (= [:user/over {:value 6}] (m/form schema))))
(testing "validation"
(is (false? (m/validate schema 6)))
(is (true? (m/validate schema 7))))
(testing "properties"
(is (= {:error/message "should be over 6"
:decode/string mt/-string->long
:json-schema/type "integer"
:json-schema/format "int64"
:json-schema/minimum 6}
(m/type-properties schema)))
(is (= {:value 6}
(m/properties schema))))))
(testing "over42"
(let [schema [Over {:value 42}]]
(testing "form"
(is (= [:user/over {:value 42}] (m/form schema))))
(testing "validation"
(is (false? (m/validate schema 42)))
(is (true? (m/validate schema 43))))
(testing "properties"
(is (= {:error/message "should be over 42"
:decode/string mt/-string->long
:json-schema/type "integer"
:json-schema/format "int64"
:json-schema/minimum 42}
(m/type-properties schema)))
(is (= {:value 42}
(m/properties schema))))))))#2020-09-2009:59ikitommi🍺 to the first who finds a valid use case for this.#2020-09-2010:01ikitommiso m/-simple-schema taks either a props map or a function of properties children => props so the props (including :type-properties) can be derived from schema properties.#2020-09-2016:51schmeeI’m trying out Malli for the first time:
(def schema [:map-of int? uuid?])
(def m {"0" "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
"1" "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"})
(m/decode schema m mt/json-transformer)
user=> {"0" #uuid "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
"1" #uuid "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}
I expected that this code would convert the keys to ints, can I modify the code in some way to make that happen?#2020-09-2016:55ikitommihi @schmee. The mt/json-transformer doesn’t transform ints from strings as ints can be presented in JSON. But - actually, this only applies to values, so I think the keys should still be converted (as all keys are strings in JSON. Could you write an issue out of this?#2020-09-2016:56ikitommito make it work today, you can use mt/string-transformer which covers that out-of-the-box.#2020-09-2016:56schmeeperfect, thanks! I’ll make an issue :thumbsup:#2020-09-2016:58schmeehttps://github.com/metosin/malli/issues/259#2020-09-2016:58ikitommire-wrote m/-predicate-schema, `m/-partial-predicate-schema` and `m/-leaf-schema` using m/-simple-schema. -39 loc.#2020-09-2016:59schmeefeel free to rename the issue to something that makes more sense to you!#2020-09-2018:05ikitommiActually, the fix was simple, we already had :map-of type transformation. Just needed add a optional argument to the mt/json-transformer that is the string-decoders, which are used for :map-of keys. The full impl looks like:#2020-09-2018:06ikitommi#2020-09-2018:08ikitommiand now:
(deftest map-of-json-keys-transform
(let [schema [:map-of int? uuid?]]
(doseq [data [{:0 "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
:1 "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}
{"0" "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
"1" "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}]]
(is (= {0 #uuid"2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
1 #uuid"820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}
(m/decode schema data mt/json-transformer))))))#2020-09-2018:09ikitommi@schmee fixed in master.#2020-09-2019:01schmeehaha, that is one fast fix, thanks! 😄#2020-09-2019:01schmeelooking forward to playing around more with Malli :thumbsup:#2020-09-2208:13steveb8nI just started using Malli for json-schema. this lib is becoming a swiss army knife for me 🙂#2020-09-2208:54ts1503Hello guys. Any ideas when a stable version of malli will arrive? I’m using the malli for a while in production but the latest release broke my app because the transitive dependency doesn’t work with Java 11 (https://github.com/borkdude/sci) 😢#2020-09-2208:56borkdude@sergey.tkachenko sci is optional with malli since a couple of releases, so if you aren't using it, it should not be a problem#2020-09-2208:57borkdudeif you are using it, then there are instructions in the README of sci what should be done with java 11#2020-09-2208:57ts1503I’m not using it directly I’m using only a malli#2020-09-2208:58borkdudethen depending on a newer version of malli should automatically fix this#2020-09-2208:58ts1503what is the latest version? I see only a 0.0.1-SNAPSHOT#2020-09-2208:59ikitommi0.0.1-20200920.124124-25#2020-09-2208:59ts1503and it worked fine for a months but after the latest release/override it broke my build#2020-09-2208:59ikitommiall SNAPSHOTs have immutable versions behind them.#2020-09-2209:00ikitommithat version has these deps:
<dependencies>
<dependency>
<groupId>borkdude</groupId>
<artifactId>dynaload</artifactId>
<version>0.1.0</version>
</dependency>
<dependency>
<groupId>borkdude</groupId>
<artifactId>edamame</artifactId>
<version>0.0.11-alpha.13</version>
</dependency>
<dependency>
<groupId>org.clojure</groupId>
<artifactId>test.check</artifactId>
<version>1.0.0</version>
</dependency>
</dependencies>#2020-09-2209:00ts1503is this version (0.0.1-20200920.124124-25) in the clojars?#2020-09-2209:01ikitommiso, you can use the “previous” SNAPSHOT too, just need to find the full version, e.g. .m2/repository/metosin/malli/0.0.1-SNAPSHOT#2020-09-2209:01ikitommiyes#2020-09-2209:01ikitommi➜ 0.0.1-SNAPSHOT ls -l
total 304
-rw-r--r-- 1 tommi staff 405 Sep 21 08:02 _remote.repositories
-rw-r--r-- 1 tommi staff 19391 Sep 21 08:02 malli-0.0.1-20200305.102752-13.jar
-rw-r--r-- 1 tommi staff 40 Sep 21 08:02 malli-0.0.1-20200305.102752-13.jar.sha1
-rw-r--r-- 1 tommi staff 1692 Sep 21 08:02 malli-0.0.1-20200305.102752-13.pom
-rw-r--r-- 1 tommi staff 40 Sep 21 08:02 malli-0.0.1-20200305.102752-13.pom.sha1
-rw-r--r-- 1 tommi staff 24437 Sep 21 07:37 malli-0.0.1-20200715.082439-21.jar
-rw-r--r-- 1 tommi staff 40 Sep 21 07:37 malli-0.0.1-20200715.082439-21.jar.sha1
-rw-r--r-- 1 tommi staff 1692 Sep 21 07:37 malli-0.0.1-20200715.082439-21.pom
-rw-r--r-- 1 tommi staff 40 Sep 21 07:37 malli-0.0.1-20200715.082439-21.pom.sha1
-rw-r--r-- 1 tommi staff 27727 Sep 20 15:43 malli-0.0.1-20200920.124124-25.jar
-rw-r--r-- 1 tommi staff 40 Sep 20 15:43 malli-0.0.1-20200920.124124-25.jar.sha1
-rw-r--r-- 1 tommi staff 1547 Sep 20 15:43 malli-0.0.1-20200920.124124-25.pom
-rw-r--r-- 1 tommi staff 40 Sep 20 15:43 malli-0.0.1-20200920.124124-25.pom.sha1
-rw-r--r-- 1 tommi staff 24437 Sep 21 07:37 malli-0.0.1-SNAPSHOT.jar
-rw-r--r-- 1 tommi staff 1692 Sep 21 07:37 malli-0.0.1-SNAPSHOT.pom
-rw-r--r-- 1 tommi staff 765 Sep 20 15:43 maven-metadata-clojars.xml
-rw-r--r-- 1 tommi staff 40 Sep 20 15:43 maven-metadata-clojars.xml.sha1
-rw-r--r-- 1 tommi staff 370 Sep 20 15:43 resolver-status.properties
#2020-09-2209:01ikitommi… should be.#2020-09-2209:01borkdudeI used to have a way of seeing these fully qualified snapshot versions on clojars. forgot#2020-09-2209:01borkdudeI usually look it up in CI when it is pushed#2020-09-2209:01ts1503there is only a snapshot#2020-09-2209:02ts1503or I’m missing something?#2020-09-2209:03borkdudebtw @ikitommi please bump dynaload to 0.2.2 :)#2020-09-2209:04ikitommiwill do + publish a new version out.#2020-09-2210:02ikitommiupdated dynaload and pushed to clojars, ping @sergey.tkachenko#2020-09-2210:02ikitommiDownloading: metosin/malli/0.0.1-SNAPSHOT/malli-0.0.1-20200922.100025-26.pom from clojars
#2020-09-2210:02ikitommithat’s the immutable version handle to it.#2020-09-2210:11ts1503Thanks#2020-09-2302:24Vincent Cantin@ikitommi This issue might be interesting for Malli, specially before the release. https://github.com/green-coder/minimallist/issues/8#2020-09-2305:16ikitommi@vincent.cantin that is the case of mixing :map and :map-of right? the issue and discussion in malli-side here: https://github.com/metosin/malli/issues/43#2020-09-2306:51ikitommidiscussion about dates, need this (date range validation) in a work project, comments welcome: https://github.com/metosin/malli/issues/49#2020-09-2306:57ikitommia good backgrounder: http://widdindustries.com/ecma-temporal-vs-java-time/#2020-09-2323:43dcjI have some code I wrote a few months back that made use of map-entries which is now gone...
IIRC, map-entries returned a collection of [key properties value-schema] ?
AFAICT in my old code, I would get this, and access the key or the properties, what should I do to move my code to entries?
What exactly is a -val-schema?#2020-09-2403:48ikitommi@dcj there is poor man's documentation about that change in CHANGELOG. The m/children returns the [key properties schema] tuple3s, so just use that instead. The new m/entries (renamed iit so it's not a silent/evil breakage) is needed to effectively use the entry properties in schema applications like value transformation and schema transformations.#2020-09-2404:02ikitommithere is a long discussion about the options and the second in the related issue (https://github.com/metosin/malli/pull/212) and it's also documented in docstrings now: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1127-L1155#2020-09-2405:48Dave SimmonsMorning - I've been using Malli for the last few month. Last night when compiling my code lein pulled down a new version. I now get a failure when trying to compile my project Exception in thread "main" Syntax error compiling at (malli/core.cljc:1164:1).#2020-09-2405:49Dave SimmonsI pulled down an earlier version of my project but in case I'd done something daft but still get the same problem. Has anyone else experienced this issue? cheers.#2020-09-2406:11ikitommi@shortlyportly i can reproduce. the new dynaload works differently (better!), will fix that.#2020-09-2406:21Dave Simmons@ikitommi - awesome - many thanks.#2020-09-2406:32ikitommi@shortlyportly should be fixed in master & [metosin/malli "0.0.1-20200924.063109-27"], ping @sergey.tkachenko too.#2020-09-2406:36ikitommihttps://github.com/metosin/malli/pull/261#2020-09-2406:35ikitommialso, sci is now loaded when it’s first used. Using preload (cljs) or direct require (cljs, graalvm, jvm) makes it eager. But: having sci on classpath and not needing it, means it wont get loaded by malli, yielding faster startup time:
(time (require '[malli.core :as m]))
"Elapsed time: 466.76624 msecs"
=> nil
(time (m/eval "(+ 1 1)"))
"Elapsed time: 1591.515317 msecs"
=> 2
(time (m/eval "(+ 1 1)"))
"Elapsed time: 0.587728 msecs"
=> 2
(time (m/eval "(+ 1 1)"))
"Elapsed time: 0.772357 msecs"
=> 2
#2020-09-2406:35ikitommiwhen clj command line tooling 3rd party library aot caching works, this can be made eager again.#2020-09-2407:01Dave Simmons@ikitommi - thank you - seems to be back and working. cheers#2020-09-2416:29dcjNext change, m/fn-schema went away?
How should I change the below....
(def registry
(merge (m/predicate-schemas)
(m/class-schemas)
(m/comparator-schemas)
(m/base-schemas)
{:zoned-date-time (m/fn-schema :zoned-date-time #'zoned-date-time?)
:local-date (m/fn-schema :local-date #'local-date?)}))
#2020-09-2416:39ikitommi@dcj there were 4 variants of the same thing, fn-schema, leaf-schema, predicate-schema, partial-predicate-schena , all replaced by m/-simple-schema. Also, there is way to reset the default registry now. But the schema:
(m/-simple-schema {:type :local-date, :pred #'local-date?})#2020-09-2416:49ikitommiit also give you a way to read the schema properties and use them for the validation or transformation:
(def Date
(m/-simple-schema
(fn [{:keys [format type min]} children]
;; check props and children here, at schema instance creation time
(let [string->date (create-formatter-somehow format)
min-date (string->date min]
{:type :local-date
:pred (fn [x] (and (instance? type x) (check-that-is-greater-than min))
:type-properties {:decode/string string->date
:error/message {:en "should be date"}
:gen/gen generator-for-date}})))
(def LocalDateThisYear
[Date {:format "YYYY-MM-DD", :min "2020-01-01", :type java.time.LocalDate}])
(m/validate LocalDateThisYear #time/local-date "2020-12-12") ; => true
(m/decode LocalDateThisYear "2020-12-12" mt/string-decoder) ; => #time/local-date "2020-12-12"#2020-09-2416:52ikitommi(def registry
(merge (m/-predicate-schemas)
(m/-class-schemas)
(m/-comparator-schemas)
(m/-base-schemas)
(m/-type-schema) ;; <--- new too
{:zoned-date-time (m/-simple-schema {:type :zoned-date-time, :pred #'zoned-date-time?})
:local-date (m/-simple-schema {:type :local-date, :pred #'local-date?})}))#2020-09-2416:54dcj@ikitommi: Thank you for all this help/context/insight!#2020-09-2419:01ikitommiadded tests and merged the lazy registries & lazy multi. Here’s the final api:
(def registry
(mr/lazy-registry
(m/default-schemas)
(fn [type registry]
;; simulates pulling CloudFormation Schemas when needed
(let [lookup {"AWS::ApiGateway::UsagePlan" [:map {:closed true}
[:Type [:= "AWS::ApiGateway::UsagePlan"]]
[:Description {:optional true} string?]
[:UsagePlanName {:optional true} string?]]
"AWS::AppSync::ApiKey" [:map {:closed true}
[:Type [:= "AWS::AppSync::ApiKey"]]
[:ApiId string?]
[:Description {:optional true} string?]]}]
(println "... loaded" type)
(some-> type lookup (m/schema {:registry registry}))))))
;; lazy multi, doesn't realize the schemas
(def CloudFormation
(m/schema
[:multi {:dispatch :Type, :lazy-refs true}
"AWS::ApiGateway::UsagePlan"
"AWS::AppSync::ApiKey"]
{:registry registry}))
(m/validate
CloudFormation
{:Type "AWS::ApiGateway::UsagePlan"
:Description "laiskanlinna"})
; ... loaded AWS::ApiGateway::UsagePlan
; => true
(m/validate
CloudFormation
{:Type "AWS::ApiGateway::UsagePlan"
:Description "laiskanlinna"})
; => true#2020-09-2515:19jeroenvandijkThank you!#2020-09-2419:03ikitommihas also the :multi key spell-checker:
(deftest multi-error-test
(let [schema [:multi {:dispatch :type}
["plus" [:map [:value int?]]]
["minus" [:map [:value int?]]]]]
(is (= {:type ["invalid dispatch value"]}
(-> schema
(m/explain {:type "minuz"})
(me/humanize))))
(is (= {:type ["did you mean minus"]}
(-> schema
(m/explain {:type "minuz"})
(me/with-spell-checking)
(me/humanize))))))#2020-09-2515:15borkdudeNow I remember how to see all the immutable SNAPSHOTS:
http://repo.clojars.org/metosin/malli/0.0.1-SNAPSHOT/
@ikitommi#2020-09-2515:33dcjHow would one specify one of these immutable snapshots in a lein project.clj?
I attempted to do exactly that earlier this week, but failed miserably....#2020-09-2515:36borkdudee.g. [metosin/malli "0.0.1-20200924.063109-27"]#2020-09-2517:06kwrooijenVery handy. Thanks!#2020-09-2515:23ikitommigreat, thanks!#2020-09-2910:05ikitommiI must have asked this before, but is there an example how to add tests “works on GraalVM too”?#2020-09-2910:16ikitommiCurrent:
(m/schema [:tuple int? int?])
; => [:tuple int? int?]
Should it be:
(m/schema [:tuple int? int?])
; => #malli/schema[:tuple int? int?]
(m/form (m/schema [:tuple int? int?]))
; => [:tuple int? int?]
GOOD:
• schemas look like schemas, not forms
BAD:
• can’t pr-str a schema and read it back with m/schema, would need malli.edn/read-string #2020-10-0223:07dcjIs there a path from Swagger/JSON-Schema -> Malli? I know there are paths the other way....
I don't expect this would be perfect and require no thought,
but imagine I encounter something described by Swagger/JSON-Schema I need/want to deal with from Clojure,
Some sort of malli translator that generated a best-effort malli schema, and also included the "and here is how to get back to JSON-Schema/Swagger" properties, and then I would edit away on the Clojure/Mailli side of the schema to make it how I want...#2020-10-0409:14ikitommi@dcj not yet, but there is https://github.com/metosin/malli/pull/211 and I recall other people trying that too. Don't have time to dig in to this yet, but would love to get that to work with malli#2020-10-0409:16ikitommisomeone was doing this for spec and I believe https://github.com/oliyh/martian has partial impl for plumatic schema.#2020-10-0700:51steveb8nQ: is there a typo in the readme in the custom registry section? :post-int#2020-10-0700:58steveb8nAnother one: is m/s-simple-schema a typo? This fn doesn’t exit#2020-10-0704:45ikitommiwelcome @kari.marttila! first one definitely a typo, could not find the latter. But PR welcome to fix any typos.#2020-10-0704:53steveb8nok. I’ll make a PR for both#2020-10-0704:55steveb8nscratch that. I don’t know what you would use to replace the second one. I’ll PR for the typo#2020-10-0704:55steveb8nhere’s the second one…
Custom registry
Example to create a custom registry without the default core predicates and with :bool and :pos-int Schemas:
(def registry
(merge
(m/class-schemas)
(m/comparator-schemas)
(m/base-schemas)
{:bool (m/-simple-schema {:type :bool, :pred boolean?})
:pos-int (m/-simple-schema {:type :pos-int, :pred pos-int?})}))
(m/validate [:or :bool :pos-int] 'kikka {:registry registry})
; => false
(m/validate [:or :bool :post-int] 123 {:registry registry})
; => true#2020-10-0704:56steveb8nm/-simple-schema doesn’t exist as a fn. I’m not sure what you would use instead#2020-10-0705:10ikitommihttps://github.com/metosin/malli/blob/master/src/malli/core.cljc#L211#2020-10-0705:10ikitommiare you using the latest version?#2020-10-0705:47steveb8nah, no. I’m not. sorry. my mistake#2020-10-0712:52ikitommiok, plan is to ship 0.1.0 tomorrow. going to clean up some internals and polish some docs, but anything relevant missing? the GraalVM tests? https://github.com/metosin/malli/pull/272#2020-10-0712:53ikitommialso, removing :list as it’s not very useful, right?#2020-10-0807:52steveb8nI haven’t used it fwiw#2020-10-0807:09ikitommi#2020-10-0808:47borkdudeWhat's the binary size?#2020-10-0809:57ikitommi18mb, with sci#2020-10-0810:00borkdudeSounds about right #2020-10-0810:00ikitommi11mb without#2020-10-0810:00borkdudeAlso sounds right!#2020-10-0817:17ikitommi[metosin/malli "0.1.0"]
• clojars ✅
• clojureverse ✅
• metosin blog ✅
• #malli ✅
• reddit ✅
• twitter ✅
• #announcements ✅#2020-10-0817:26ikitommiok, thank you everyone, Malli is officially out! if there are more places to share, please do.#2020-10-0818:03dcjCongrats! And thank you for your amazing work on this!#2020-10-0906:49ikitommithanks!#2020-10-0817:58dharriganIs it still alpha?#2020-10-0818:08borkdude@dharrigan I think it went from pre-alpha to alpha now. So in Clojure world, it's production time? 😆#2020-10-0818:09dharriganYeah, that's what I think. I think, given what happened a few weeks/months ago, about the push from the core team to move away from eternal alpha to better versions, and given that malli is now out, then perhaps alpha isn't the right term here?#2020-10-0818:09dharriganIt's not even beta anymore#2020-10-0818:11borkdudeWhat move away from eternal alpha? spec has been in alpha since it's existence#2020-10-0818:13dharriganYeah, there are exceptions of course#2020-10-0818:14dharrigan#2020-10-0818:14dharriganthere you go#2020-10-0818:15dharriganAfter that blog post on Febrary, quite a few libraries moved away from alpha, the eternal alpha.#2020-10-0818:16dharriganFrom first-hand experience trying to introduce Clojure (and libraries into the application), seeing things as alpha can be off-putting to others in the teams.#2020-10-0818:16dharrigan.#2020-10-0820:45katoxIt's Clojure we use it in production since June. And it's great.#2020-10-0821:06dharriganI've been using malli now for a few months, stable, featureful and works brilliantly 🙂#2020-10-0821:17schmee@ikitommi is it of interest to have a Schema -> Postgres DDL transformation in Malli? 🙂#2020-10-0821:21dcjI'm interested.... I have a simple prototype that handles only what I need, and "works for me"....#2020-10-0822:01kwrooijenI actually wrote a library that does that https://github.com/kwrooijen/gungnir#2020-10-0822:01kwrooijenIn combination with HoneySQL / next.jdbc#2020-10-0822:02kwrooijenIt allows you to define your table structure, with transformations / validations, using Malli#2020-10-0822:09dcj@UG9U7TPDZ Wow, that looks cool! I will definitely try that out.#2020-10-0823:19macIs it correctly understood that malli does not support sets as predicates? if so is there then an alternative?#2020-10-0900:06schmee@mac the equivalent is [:enum ...]#2020-10-0906:38mac@schmee Ah, thanks.#2020-10-0906:55ikitommi@dharrigan could have been a non-alpha, but was busy getting it out, so just alpha now. Grand goal is to get 1.0.0 out as soon as possible, after the final pieces (sequences, parsing, functions, default options) have been implemented, might effect the public api so that need to bump MAJOR. Most likely something that is easily migrated from pre-1.0.0.#2020-10-0907:00ikitommi@mac there is a special type-schemas mechanism which would allow using using plain sets, but having too much shortcuts might make things harder to understand. There is already a such type shortcut for regexs, not sure if that was the right call:
#"\d+kikka" ;; using the type-shortcut
[:re #"\d+kikka"] ;; explicit
sets - besides :enum, there is :fn for any function.
[:enum "a" "b" "c"]
[:fn #{"a" "b" "c"}]
:enum doesn’t do type inferring atm, so you can hint the type so that JSON Schema, value transformation etc. work correctly:
[:and keyword? [:enum :kikka :kukka]]
#2020-10-0907:02dharrigan@ikitommi fantastic! Thank you for the update (and also the libraries, reitit and malli - super awesome sauce!)#2020-10-0907:23mac@ikitommi Thanks, the reg-ex shortcut looks pretty idiomatic to me.#2020-10-0909:09borkdudeAre there any libs like cli-matic that use malli instead of spec for arg parsing/validation?#2020-10-0912:18Toni VanhalaToday was supposed to be ClojuTRE 2020, but what we got was t-shirts with malli validation errors. Just put these fun little items to Metosin shop:
https://shop.spreadshirt.net/metosin/#2020-10-0917:57Lucas Félixhi! - sorry for my english - is there something like spec functions (https://clojure.org/guides/spec#_specing_functions) in malli? if not, what’s the motivation?#2020-10-0918:22borkdude@lfelixsampaio https://github.com/metosin/malli/issues/125#2020-10-0918:24Lucas Félixthanks! @U04V15CAJ#2020-10-0922:56steveb8n@ikitommi I’m curious why you’ve stopped mentioning the multi-tenant validation capabilities of Malli. for me, this was the original reason for using it and the registry design is very well done#2020-10-1007:48borkdude@ikitommi I'm reading through the Malli README and found a couple of spelling improvements:
Homogenous -> Homogeneous#2020-10-1007:49borkdudeAnd this sentence:
> You can also decomplected maps keys and values using registry references.
seems to be grammatically a bit off#2020-10-1008:13ikitommi@borkdude fixes are most welcome.#2020-10-1008:13borkdudeJust passing it here, do what you want with it :)#2020-10-1008:15ikitommiwill fix#2020-10-1008:17ikitommifixed#2020-10-1008:18ikitommitoyed with the declarative schema transformations, which might be useful when defining schemas in EDN:
(require '[malli.core :as m])
(require '[malli.util :as mu])
(require '[malli.error :as me])
(def registry (merge (m/default-schemas) (mu/schemas)))
(def XZ
(m/schema
[:select-keys
[:merge
[:map [:x int?]]
[:map [:y int?]]
[:map [:z int?]]]
[:x :z]]
{:registry registry}))
XZ
; [:select-keys
; [:merge
; [:map [:x int?]]
; [:map [:y int?]]
; [:map [:z int?]]]
; [:x :z]]
;; get the effective schema
(m/deref XZ)
; [:map [:x int?] [:z int?]]
;; internally uses the pre-computed effective schema
(-> XZ
(m/explain {:x 1})
(me/humanize))
; {:z ["missing required key"]}#2020-10-1008:21borkdudeI was wondering, does malli also do the destructuring that spec does e.g. on a sequential regex schema? (s/cat etc)#2020-10-1008:21borkdudeor s/or, etc#2020-10-1008:22borkdudesomeone should probably write a comparison/migration page for malli <-> spec :)#2020-10-1008:22ikitomminot yet, the internal api works, but not integrated into malli.core: https://github.com/metosin/malli/issues/180#2020-10-1008:23ikitommi… and https://github.com/metosin/malli/issues/241 after that.#2020-10-1008:24borkdudeexactly! thanks!#2020-10-1018:53ziltiWhen creating a validator, how do I use non-core functions in a :fn? Specifically, I want to use clojure.string/blank?#2020-10-1019:48ikitommi@zilti :fn takes any function as unquoted, so [:fn clojure.string/blank?]. They don't serialize correctly, but work on the same runtime. If you use sci, you need to add custom bindings for the function for all runtimes. I believe str/blank? is part of sci default bindings, so [:fn 'str/blank?] should work too.#2020-10-1019:49ikitommialso, you can say [:string {:min 1}].#2020-10-1019:49borkdudeclojure.string/blank? works in sci by default#2020-10-1019:49ziltiOh, that :min seems to be the most straightforward for my usecase, I'll use that!#2020-10-1019:52ikitommi:string has good default error messages: https://github.com/metosin/malli/blob/master/test/malli/error_test.cljc#L262-L283#2020-10-1110:21Lucy WangFixed a bunch of typos https://github.com/metosin/malli/pull/275#2020-10-1111:00ikitommimerged, thanks!#2020-10-1112:10ikitommiwould like to make sci explicitly optional. either via a flag (non-breaking) or via explicit option (breaking, but for the better): https://github.com/metosin/malli/issues/276. Two days ago would have just done the latter, but now malli is released and goal has been not to break things. What do you think? save the breaking change for 1.0.0? just do it? something else?#2020-10-1112:13borkdude@ikitommi Maybe sci could be one of many possible evaluators?#2020-10-1202:33steveb8nI think a breaking change for the 1.0 release is ok. We know you plan to accrete only from 1.0#2020-10-1202:34steveb8noptional sci is great for the browser use case. making that the default is a good idea imho#2020-10-1202:34steveb8nI’m gonna use it server side also#2020-10-1205:48ikitommigood reason not yet to read sci-powered schemas from untrusted sources: https://github.com/borkdude/sci/issues/348#2020-10-1205:48ikitommithat's now linked in Malli Readme.#2020-10-1112:13borkdudelike clojure.core/eval is another one#2020-10-1112:13borkdudeand maybe the user should specify that#2020-10-1112:14borkdudebreaking would be ok I think#2020-10-1112:14borkdudesince the project is explicitly alpha#2020-10-1112:14ikitommiyes, the option1 would allow that:
(require '[malli.sci :as ms])
(require '[malli.core :as m])
(def options
{:evaluator (ms/evaluator)
:registry (ms/default-registry)})
(def Schema (m/schema [:fn '(fn [x] (string? x))] options))
(m/validate Schema "kikka")
; => true#2020-10-1112:15ikitommi.. would also remove need of :preloads etc, as you actually need to require the code to make it work.#2020-10-1112:15borkdudeI think that's reasonable as long as it's documented well#2020-10-1112:16borkdudeand keep a list of breaking changes in CHANGELOG.md#2020-10-1112:18borkdudemalli.sci would still require sci for you right. so then there's no need for preloads#2020-10-1112:19ikitommialso, currently you can swap the default registry using the JVM/clj-compiler options, but not the default options. if you want to enable sci globally, that should be changed to swap the default options. I did a spike on that, but had too much open issues to think that through. Now, it seems like it would have been a right call.#2020-10-1112:19ikitommihttps://github.com/metosin/malli/pull/235#2020-10-1112:21ikitommithat would be big breaking change, but something that could be documented and migrated easily too.#2020-10-1112:21ikitommimaybe all roads lead for quick 1.0.0 🙂#2020-10-1112:22ikitommithanks for you thoughts on this.#2020-10-1114:00motformHi! does anyone have any experience using malli to validate/define the re-frame app-db?#2020-10-1122:13steveb8n@love.lagerkvist I’m doing this for forms currently https://github.com/stevebuik/fork-malli-ideas#2020-10-1122:14steveb8nworks great. have not moved the app-db level validations from spec to Malli yet but I certainly plan to do so#2020-10-1203:55Lucy Wang@love.lagerkvist I'm using re-frame.core/reg-global-interceptor to verify the app-db matches my schema in every change, and prints an console warning if it doesn't conform. But re-frame.app-db is just a normal map (live inside an reagent.atom), so it's nothing special when using it with malli IMO.#2020-10-1205:35ikitommimerged #277, non-breaking disabling & configuration of sci. Will make a 1.0.0 issue with suggestion to make swappable & explicit evaluator. Thanks @borkdude @steveb8n#2020-10-1207:00borkdude@ikitommi note that termination safe does have impact on performance of realizing seqs (since it’s checked). I’m not sure if the issue about execution time is fundamentally solvable.#2020-10-1208:44ikitommiIt is one of the harder problems in comp.sci ;) With Malli & sci, I concider safety as more important. Following the issue & pr on sci-side, happy to incorporate anything that is found for this.#2020-10-1208:51borkdudeI think the fundamental solution would be to run the sci expression in a thread on the JVM and kill it if it takes too long or in a webworker in the browser and do the same.#2020-10-1208:51borkdudeMaybe I should drop the termination-safe option as well - not sure.#2020-10-1207:54motform@wxitb2017 that sounds about right, I’ll do that. However, I was also thinking if it was possible to co-locate/generate/properly infer an inital app-db directly from malli. It would be really nice to have a single source of truth, kind of like a reitit router with views, coercion and controllers. (it’s probably super possible and I have to dig more into malli or a very bad idea)#2020-10-1215:15ikitommiYou could have a Malli schema defined for the app-db, with default and create the initial state from those, e.g.
(defn create [?schema]
{:validator (m/validator ?schema)
:explainer (m/explainer ?schema)
:initial (m/decode ?schema nil (mt/default-value-transformer))})
(create [:map {:default {}}
[:user [:map {:default {}}
[:first-name [:string {:min 1, :default ""}]]
[:last-name [:string {:min 1, :default ""}]]]]])
;{:validator #object[...],
; :explainer #object[...],
; :initial {:user {:first-name "", :last-name ""}}}
inferring is powerful, but not fully accurate, e.g. is something a vector of things of a tuple of always 2.#2020-10-1215:16ikitommi(the mt/default-transformer could have an option to create empty maps by default…)#2020-10-1307:26ikitommihttps://github.com/metosin/malli/pull/278, added :key and :defaults to mt/default-value-transformer to make it easier to create empty/default values with it:
(m/decode
[:map
[:user [:map
[:name :string]
[:description {:ui/default "-"} :string]]]]
nil
(mt/default-value-transformer
{:key :ui/default
:defaults {:map (constantly {})
:string (constantly "")}}))
; => {:user {:name "", :description "-"}}#2020-10-1307:27ikitommithe values in :defaults can access the Schema instance, so can use any information from it to create the default, e.g. schema -> default function.#2020-10-1308:36dangercoderIs it possible to override the default error message for :re?#2020-10-1309:06kwrooijen[:re {:error/message "Invalid email"}
#"
Something like this?#2020-10-1309:32dangercoderI would like it to be general#2020-10-1309:32dangercoderLike, for all regexp errors I apply a custom message#2020-10-1309:36kwrooijenAh like that. I'm not familiar with a solution for that#2020-10-1310:58ikitomminot a good way to do that, but one can override the :errors from malli.error like this:
(-> #"\d+"
(m/explain "kikka")
(me/humanize {:errors {:re {:error/message "not a number"}}}))
; => ["not a number"]#2020-10-1313:26dangercoderMy goal is to have an error message like this:
"Value: %s does not comply with regexp: %s" 🙂#2020-10-1313:34ikitommimaybe:
(defn regex-error [{:keys [value schema]} _]
(str "Value: " value " does not comply with regexp: " (first (m/children schema))))
(-> #"\d+"
(m/explain "kikka")
(me/humanize {:errors {:re {:error/fn regex-error}}}))
; => ["Value: kikka does not comply with regexp: \\d+"]#2020-10-1313:40dangercoderI'll check it out. Thanks for the inspiration @ikitommi#2020-10-1309:15raymcdermottI am hitting a dumb problem with references#2020-10-1309:15raymcdermott(ns data.recursive-case
(:require [malli.util :as mu]))
(def asset
[:map
[:asset/id string?]
[:asset/status [:enum :initialized :accepted :active :revoked :failed]]
[:asset/asset {:optional true} asset]])
(def org-asset
(mu/merge
asset
[:map
[:asset/org :org]]))
(def user-asset
(mu/merge
asset
[:map
[:asset/user :user]]))#2020-10-1309:16raymcdermottasset works OK but merge fails#2020-10-1309:16raymcdermottExecution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema #object[clojure.lang.Var$Unbound 0x4bcbe824 "Unbound: #'data.recursive-case/asset"]}#2020-10-1309:17raymcdermottpretty sure I've misunderstood something to do with registries so would appreciate if anyone can suggest what's up this#2020-10-1309:27kwrooijenI think this line is wrong in the asset def?
[:asset/asset {:optional true} asset]]#2020-10-1309:27kwrooijenYou're referring to asset, but it's in the definition itself#2020-10-1309:28kwrooijenJudging from the naming, you actually want it to be recursive?#2020-10-1309:33kwrooijenYou might need to add asset into a registry which you actually want to use it in. e.g. something like this:
{::asset
[:map
[:asset/id string?]
[:asset/status [:enum :initialized :accepted :active :revoked :failed]]
[:asset/asset {:optional true} [:ref ::asset]]]}
https://github.com/metosin/malli#recursive-schemas#2020-10-1312:54raymcdermottyes ... I figured it was a registry thing. Thanks for the advice#2020-10-1311:52katox@ikitommi I have been bitten by round-trip java.time.Instant -> string -> malli -> #inst. The problem manifests in java9+ because of increased precision from miliseconds to microseconds. I noticed there is a bigger plan for time handling code but in the meantime I patched the string->date transformer. Should I do a PR for that? See https://github.com/metosin/malli/compare/master...katox:timestamp-frac-precision#2020-10-1313:24miikkaSeems like a good idea (as long as the code still works on Java8 to the extent that it can work)#2020-10-1407:50katoxI tested the change in both j8 and j11. It works the same. Issuing PR.#2020-10-1315:34shemi have JSON data that has entries like "Start":new Date(1413147600000) . clojure JSON parsers (at least Cheshire and Jsonista) choke on this. i wonder if it's possible to use malli.core/decode with the :enter interceptor to parse this and transform it to "Start": 1413147600000 ? my first attempt produces an exception#2020-10-1315:34shem#2020-10-1318:19sparkofreasonSchema inference is awesome going to save me a tone of time. Before I start reinventing a wheel, is anybody working on transforming malli schema to Datomic schema?#2020-10-1516:23ikitommihaven’t seen anything for this, looking forward to your example 🙂#2020-10-1714:24sparkofreasonLooks to be pretty straightforward. I've done this with spec before. There were two headaches there, one being the lack of a public data representation for specs, second being opaque predicates. In the latter case we used test.check to generate examples and tried to infer from that. I think you do the same thing here, but leveraging malli's inference.#2020-10-1515:49ikitommi@raymcdermott noticed that none of the malli.util helpers deref the refernce schemas, wrote an https://github.com/metosin/malli/issues/281. Before that, to use registry references, one needs to deref those manually, one option being:
(require '[malli.core :as m])
(require '[malli.registry :as mr])
(require '[malli.util :as mu])
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
{:asset/id string?
:asset/status [:enum :initialized :accepted :active :revoked :failed]
:asset/asset [:map
:asset/id
:asset/status
[:asset/asset {:optional true} [:ref :asset/asset]]]}))
(def org-asset
(mu/merge
(m/deref :asset/asset)
[:map [:asset/org [:= "org"]]]))
org-asset
;[:map
; [:asset/id :asset/id]
; [:asset/status :asset/status]
; [:asset/asset {:optional true} [:ref :asset/asset]]
; [:asset/org [:= "org"]]]
(def user-asset
(mu/merge
(m/deref :asset/asset)
[:map [:asset/user [:= "user"]]]))
user-asset
;[:map
; [:asset/id :asset/id]
; [:asset/status :asset/status]
; [:asset/asset {:optional true} [:ref :asset/asset]]
; [:asset/user [:= "user"]]] #2020-10-1515:51ikitommialso, the declarative :merge would allow registries like this:
{:asset/id string?
:asset/status [:enum :initialized :accepted :active :revoked :failed]
:asset/asset [:map
:asset/id
:asset/status
[:asset/asset {:optional true} [:ref :asset/asset]]]
:asset/org-asset [:merge :asset/asset [:map [:asset/org [:= "org"]]]]
:asset/user-asset [:merge :asset/asset [:map [:asset/user [:= "user"]]]]}#2020-10-1516:21ikitommi@shem, you example doesn’t seem legit syntax. But for JSON, you need to defin :decode/json key. Something like:
(m/decode
[:map
[:Duration string?]
[:Start {:decode/json (fn [x] (mt/-string->long (last (re-find #"(\d+)" x))))} string?]]
{:Duration "kikka"
:Start "new Date(1413147600000)"}
(mt/json-transformer))
;{:Duration "kikka"
; :Start 1413147600000}#2020-10-1516:23ikitommiif your new Date is coming from js, it’s a string of Mon Oct 13 2014 00:00:00 GMT+0300 (Eastern European Summer Time) {}, and you need to run it via some date-transforming fn. hope this helps.#2020-10-1516:37ikitommiWould it be a breaking change if m/deref was recursive and would would return the input schema in case it was not a RefSchema? I would like it to be:
(require '[malli.core :as m])
(m/schema [:schema [:schema [:schema [:map [:x int?]]]]])
; => [:schema [:schema [:schema [:map [:x int?]]]]]
(m/deref (m/schema [:schema [:schema [:schema [:map [:x int?]]]]]))
; => [:map [:x int?]]
, instead of:
(m/deref (m/schema [:schema [:schema [:schema [:map [:x int?]]]]]))
; => [:schema [:schema [:map [:x int?]]]]
#2020-10-1516:38ikitommialso:
(m/deref [:map [:x int?]])
; => [:map [:x int?]]
instead of:
(m/deref [:map [:x int?]])
; Execution error (IllegalArgumentException)
; No implementation of method: :-deref of protocol: #'malli.core/RefSchema found for class: malli.core$_map_schema$reify$reify__4587
#2020-10-1517:25eraadHi guys! I’m looking for any pointers on how to transform keys (ie snake to kebab) in nested schemas. I have the following schema:
(def Invoice
[:map {:registry {::address Address
::tax-line TaxLine}}
[:issue_date [string? epoch]]
[:series string?]
[:sequence [string? {:decode/string mt/-string->long}]]
[:currency [string? (lookup-ref :currency/code)]]
[:customer_legal_name string?]
[:customer_tax_id string?]
[:customer_tax_id_type [string? (lookup-ref :tax-id/type)]]
[:customer_email {:optional true} string?]
[:customer_address {:optional true} ::address]
[:tax_lines [:vector ::tax-line]]
[:subtotal [string? {:decode/string mt/-string->double}]]
[:tax [string? {:decode/string mt/-string->double}]]
[:total [string? {:decode/string mt/-string->double}]]])
When i do:
(defn coerce-request
[schema req]
(if (m/validate schema req)
(m/decode schema
req
nil
(mt/transformer
mt/json-transformer
mt/string-transformer
{:name :epoch}
{:name :lookup-ref}
(mt/key-transformer {:decode k/->kebab-case})))
(-> schema
(m/explain req)
(me/humanize))))#2020-10-1517:25eraadcustomer_address map keys are not affected#2020-10-1518:06ikitommi@eraad decoding is a process of taking an invalid value and transforming it to a valid (EDN) value. Because of this, the key-transformer runs the transformations on :enter phase. All the top-level map keys are transformed into invalid (kebab)-values and the :customer_address gets a value of :customer-value. After this, the keys don’t match the schema and the child decoders are no-op.#2020-10-1518:08ikitommiif you need to transform values out from the valid (EDN) definitions, you can use m/encode. the key-transformer runs in :leave phase there, in your case as the last thing as it’s also last transformer in the chain.#2020-10-1518:09eraadHi @ikitommi thank you for the response :thumbsup:#2020-10-1518:10ikitommihere’s a minimal sample:
(m/decode
[:map {:registry {::address [:map [:street string?]]}}
[:customer_address ::address]]
{:customer_address {:street "hämeenkatu"}}
(mt/transformer
(mt/key-transformer {:decode #(-> % name str/upper-case keyword)})))
; => {:CUSTOMER_ADDRESS {:street "hämeenkatu"}}
(m/encode
[:map {:registry {::address [:map [:street string?]]}}
[:customer_address ::address]]
{:customer_address {:street "hämeenkatu"}}
(mt/transformer
(mt/key-transformer {:encode #(-> % name str/upper-case keyword)})))
; => {:CUSTOMER_ADDRESS {:STREET "hämeenkatu"}}#2020-10-1518:12eraadNice, thanks!#2020-10-1518:13eraadI’m really loving Malli, tried Spec+Spec-tools for a bit but then tried Malli and its great#2020-10-1518:13ikitommiin your case, you can write your own rename-keys like this (running on :leave in decode):
(m/decode
[:map {:registry {::address [:map [:street string?]]}}
[:customer_address ::address]]
{:customer_address {:street "hämeenkatu"}}
(mt/transformer
{:decoders {:map {:leave (mt/-transform-map-keys #(-> % name str/upper-case keyword))}}}))
; => {:CUSTOMER_ADDRESS {:STREET "hämeenkatu"}}#2020-10-1518:15ikitommi“on :decode, for :maps run this function f on :leave”. transformers are quite simple in the end 😉#2020-10-1518:15eraadCoo, I was not being aware of :enter and :leave, that’s the missing part#2020-10-1615:23ikitommiDid a PR on making :schema and :ref schemas behave better: https://github.com/metosin/malli/pull/282. It has a BREAKING CHANGE as m/deref would be recursive and does not throw.#2020-10-1615:26ikitommithese work after the PR:
1. mu/merge and mu/union with references:
(mu/merge
[:schema [:map [:x int?]]]
[:map [:y int?]])
; => [:map [:x int?] [:y int?]]
2. mu/subschemas with both :ref & :schemas:
(mu/subschemas
[:ref {:registry {"Address" [:map
[:street :string]
[:address [:ref "Address"]]
[:neighbor [:ref "Neighbor"]]]
"Country" [:map [:name [:= "finland"]]]
"Neighbor" [:ref "Address"]}}
"Address"])
;[{:path [], :in [], :schema [:ref {:registry {"Address" [:map
; [:street :string]
; [:address [:ref "Address"]]
; [:neighbor [:ref "Neighbor"]]]
; "Country" [:map [:name [:= "finland"]]]
; "Neighbor" [:ref "Address"]}}
; "Address"]}
; {:path [0 0], :in [], :schema [:map
; [:street :string]
; [:address [:ref "Address"]]
; [:neighbor [:ref "Neighbor"]]]}
; {:path [0 0 :street], :in [:street], :schema :string}
; {:path [0 0 :address], :in [:address], :schema [:ref "Address"]}
; {:path [0 0 :neighbor], :in [:neighbor], :schema [:ref "Neighbor"]}]#2020-10-1615:28ikitommiif you are in business of generating forms from schemas, mu/subschemas is your best friend.#2020-10-1615:29ikitommim/walk can be configured with ::m/walk-refs and ::m/walk-schema-refs to walk over none, some of all references. Does not StackOverFlow, walks a given reference just once. cheers.#2020-10-1621:02dangercoderis there some kind of way to speed up m/validate?
I have a real edge case, I automatically generated a schema for a structure that potentially can contain 1294 fields (a tree structure...) when I ran validate it took a few seconds on a empty map.#2020-10-1621:03dangercoderwould it run quicker if I used a registry for things that are "common"?#2020-10-1621:04borkdudeI think malli supports something like lazy validation and/or lazy schemas - I've seen this being mentioned in a conversation with ikitommi and jeroenvandijk#2020-10-1623:41jeroenvandijk@jarvinenemil not sure if it is still idiomatic Malli and it can be more optimized, but here is how you could do lazy loading (see lookup-type) https://github.com/jeroenvandijk/aws.cloudformation.malli/blob/master/src/adgoji/aws/cloudformation/malli/validation.clj#2020-10-1701:39ikitommi@jarvinenemil are you using m/validator or m/validate? the latter is a convenience fn for one-shot things: if given schema syntax, for every call, it parses it, creates the Schema instance tree and creates a pure validator for it and runs it once and throws it away. m/validator returns that pure validation function of x -> boolean , so if you can keep that, you pay for the parsing & optimizing just once.#2020-10-1701:40ikitommisame applies to m/explain vs m/explainer, m/decode vs m/decoder etc.#2020-10-1701:41ikitommihttps://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1079-L1085#2020-10-1701:45ikitommiat best, given a deeply nested schema with 1k fields, the m/decoder can return a indentity function if there are no decoding functions mapped to any of the fields or structure in general. Can't get much faster than that 😉#2020-10-1709:55dangercoder@ikitommi I did a one off using validate. validator was just what I was looking for, thanks!#2020-10-1714:15ikitommicurious about the perceived perf gains, hopefully many orders of magnitude..#2020-10-1715:02dangercoderUsing validate
(cc/quick-bench (m/validate schema {}))
Evaluation count : 6 in 6 samples of 1 calls.
Execution time mean : 21.971360 sec
Execution time std-deviation : 2.848014 sec
Execution time lower quantile : 20.065855 sec ( 2.5%)
Execution time upper quantile : 26.840086 sec (97.5%)
Overhead used : 7.984810 ns
Using validator
(cc/quick-bench (validator {}))
Evaluation count : 31504632 in 6 samples of 5250772 calls.
Execution time mean : 12.055681 ns
Execution time std-deviation : 0.575152 ns
Execution time lower quantile : 11.599548 ns ( 2.5%)
Execution time upper quantile : 12.803005 ns (97.5%)
Overhead used : 7.984810 ns#2020-10-1715:19ikitommithat's 9 orders of magnitude 😮😮😮 . Are you starting a power plan while creating the schema as it takes 20 seconds???#2020-10-1715:21ikitommianyway, good reason to cache the validator :)#2020-10-1715:21dangercoderI converted a excel-sheet from business people to a malli schema so I can convert the malli schema to a json-schema that can be used from several languages 😄#2020-10-1715:28dangercoderI anonymized the fields and havent added regexp yet but just to give u an idea on how big this thing is. @ikitommi
I did not decide this, the real world is sometimes a bit messy 😄.#2020-10-1714:16ikitommiWIP, https://github.com/metosin/malli/pull/283:
(def registry (merge (m/default-schemas) (mu/schemas)))
(def Merged
(m/schema
[:merge
[:map [:x :string]]
[:map [:y :int]]]
{:registry registry}))
Merged
;[:merge
; [:map [:x :string]]
; [:map [:y :int]]]
(m/deref Merged)
;[:map
; [:x :string]
; [:y :int]]
(m/validate Merged {:x "kikka", :y 6})
; => true#2020-10-1714:19ikitommijust :merge , :union and :select-keys now, should be easy to add more, with the util helpers. current Schema impls in malli.util:
(defn -merge [] (-util-schema {:type :merge, :f (-reducing merge)}))
(defn -union [] (-util-schema {:type :union, :f (-reducing union)}))
(defn -select-keys [] (-util-schema {:type :select-keys, :min 2, :max 2, :f (-applying select-keys)}))#2020-10-1717:04jfntn👋 I’m looking for a way to use an accumulator during a schema walk.#2020-10-1717:04jfntnThat’s easy to hack from the outside in with say an atom but I was wondering if there is cleaner way to do this from the inside out instead?#2020-10-1717:05jfntnIt seems like it’d be possible to reify a Walker that expects the walk fn to return [schema context] and merges the returned context into the options to pass to the next step…#2020-10-1717:40ikitommiyes, you should implement a custom Walker. find-first might be a good example for that: https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L37-L51#2020-10-1717:41ikitommior: subschemas: https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L150-L163#2020-10-1718:05jfntnThanks for the pointers!#2020-10-1718:06jfntnI guess doing this without some atom would require some kind of state monad so I think I’ll stick with your examples 😂#2020-10-1718:21ikitommiif you get a generic accumulating walker, might be a good example in https://github.com/metosin/malli/blob/master/docs/tips.md.#2020-10-1914:59Vincent CantinThere might be a way to bridge Specter and Malli for that purpose.#2020-10-1718:45mike_ananev@ikitommi can I have docstring for spec in swagger output?#2020-10-1719:21ikitommi@mike1452 what kind of docstring are you looking for? :title, :description and :default are automatically copied to both JSON Schema and Swagger.#2020-10-1719:23ikitommiJSON Schema references#2020-10-1807:21mike_ananev@ikitommi thanks! this example is what I need.#2020-10-1814:10ikitommiI think declarative schema transformations is done now: https://github.com/metosin/malli/pull/283.#2020-10-1906:09ikitommi🎉#2020-10-1907:17dharriganFantastic! Going to try these out now!#2020-10-1907:48ikitommihttps://malli.io/?value=%7B%3Atype%20%22Cat%22%2C%20%3Aname%20%22Viivi%22%2C%20%3AhuntingSkill%20%3Aadventurous%7D&schema=%5B%3Aschema%20%7B%3Aregistry%20%7B%22Pet%22%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atype%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20string%3F%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Cat%22%20%5B%3Amerge%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Pet%22%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atype%20%5B%3A%3D%20%22Cat%22%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3AhuntingSkill%20%5B%3Aenum%20%7B%3Adescription%20%22The%20measured%20skill%20for%20hunting%22%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Aclueless%2C%20%3Alazy%2C%20%3Aadventurous%2C%20%3Aaggressive%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Dog%22%20%5B%3Amerge%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Pet%22%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atype%20%5B%3A%3D%20%22Dog%22%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3ApackSize%20%5B%3Aint%20%7B%3Amin%200%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Adefault%200%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Adescription%20%22the%20size%20of%20the%20pack%20the%20dog%20is%20from%22%7D%5D%5D%5D%5D%7D%7D%0A%20%5B%3Amulti%20%7B%3Adispatch%20%3Atype%7D%20%22Cat%22%20%22Dog%22%5D%5D#2020-10-1913:32ikitommiwhat a brilliant/horrible idea: add optional :object schema, which allows maps to be described using clojure maps. Kinda like data-specs but for malli:
[:object
{"name" [:required :string]
"age" [:required :number :positive :integer]
"email" [:string :email]
"website" [:string :url]
"createdOn" [[:date {:default "2020-18-10"}]]
"address" [:object
{"street" [[:string {:min 1}]]
"zip" [:int]}]}]#2020-10-1913:35ikitommior:
[:object
{"name" [:and :required :string]
"age" [:and :required :number :positive :integer]
"email" [:and :string :email]
"website" [:and :string :url]
"createdOn" [:date {:default "2020-18-10"}]
"address" {:street [:string {:min 1}]
:zip :int}}]#2020-10-1914:17borkdudea bit like Schema, but more EDN-like#2020-10-1914:37ikitommi☝️. simple formats like Schema and data-spec are kinda easy, but not simple: the core utilties (`select-keys`, assoc etc.) almost work, unless you have a wrapper for keys like ds/opt or s/optional-key in case they don’t. Also, value wrappers are needed to add visible properties/meta-data to schemas. And there is no order for keys. But, super nice for many things like defining inlined route parameters:
["/plus"
{:get {:summary "plus with spec query parameters"
:parameters {:query {:x int?, :y int?}}
:responses {200 {:body {:total int?}}}
:handler handle-plus}]#2020-10-1914:40ikitommiif there was a litemalli coercion in reitit, one could swap the data-spec apps almost 1:1 to it.#2020-10-1914:47pithylessI prefer #1 to #2; the latter is making nested map notation implicit and special for :object and I don't know if it's worth it.#2020-10-1914:51pithylessThat reitit example could just as easily be solved by a utility function that takes the concise nested map notation and converts it to a tree of :object#2020-10-1914:52borkdudesomeone should write malli-tools ;)#2020-10-1914:52ikitommi:grinning_face_with_one_large_and_one_small_eye:#2020-10-1914:58pithylessDoes :object offer different semantics or tradeoffs than :map? Or would it deref/merge down to a :map schema? Or would we have two things that are the same but different? :]#2020-10-1914:59pithylessTo be clear, I prefer writing the map notation and was always confused why :map used vectors; but that ship has sailed... or has it?#2020-10-1915:01borkdudewe're in a dynlang, we can do everything we want ;)#2020-10-1915:03pithylessI'm thinking :object could be just a nice alternative syntax that would "compile down" to a :map. But maybe I'm just missing the point. :)#2020-10-1915:04ikitommihttps://github.com/metosin/malli/issues/286#2020-10-1915:06borkdudewhy it is called object though?#2020-10-1915:06borkdudelike JavaScript object?#2020-10-1915:08borkdudeI was thinking like: wow, is this a schema to describe Java objects like bean-stuff?#2020-10-1915:08borkdudeanyone not working on front-ends might think the same#2020-10-1915:09ikitommivector syntax for :map allows simple way to define properties for it and retains the order of keys. But yes, we can add more schemas to build things in different ways. Could have also optional :`json-schema` that allows any JSON Schema in it etc.#2020-10-1915:13ikitommithe name could be anything, just flushing stuff from my brain (after a vacation).#2020-10-1915:19pithylessIMO, it would be confusing to see both :map and :object somewhere in my system output and they were semantically the same thing (a representation of known key-value mapping); so the way I see it:
1. if we want an alternative writing notation, why not just make it a utility function? (instead of introducing a new kind of schema)
2. if we want an alternative notation that is a data-literal (utility functions need runtime support), then we could introduce an alternative name (like :object ); but it could just as easily be :kv or :map2 ; or even why not extend existing :map to support both kinds of notations (vectors or maps)
3. if this is something similar to :map but offers possibly different semantics, than currently those semantics aren't clear to me; otherwise it'd be great if we didn't have two things that represent essentially the same thing (this would make merging, etc. more complicated to deal with later on)#2020-10-1915:21pithyless(This is also why I like having :map and :map-of , because they represent different semantics; even though they are using the same internal datastructure)#2020-10-1915:38ikitommiRelated to 2, I think they should be kept separate (if implemented). If :map supported both syntax, there would be ambiguity issues:
[:map
{:id :int
:title :string}]
is that empty map with props? or map with no props but with keys?#2020-10-1915:43ikitommiIf I would implement that today, it would be a separate ns, malli.object etc, with a Schema impl, a helper and a extra registry snipplet so one can merge the schemas into a registry easily:
(require '[malli.core :as m])
(require '[malli.object :as mo])
(def registry (merge (m/default-schemas) (mo/schemas)))
(m/schema
[:object
{:id int?
:address [:object {:street string?}]}]
{:registry registry})#2020-10-2112:28salva-xfHello, I found malli a couple of days ago and I'm liking it although I don't know if I really understand it well, is malli suitable for defining state machines for example? In general, it is suitable for defining data structures to go through and generate code? generate the models for the orientdb and the datascript for example? Would it be suitable for, for example, defining IFML structures and generating the user interfaces from them? These are the questions that I try to answer while I see the documentation, thank you very much#2020-10-2116:19Lucy WangMalli could do most what spec could do, data validation, data transform/coercion, data generation etc. This might be a good refererence https://www.pixelated-noise.com/blog/2020/09/10/what-spec-is/#2020-10-2119:36ikitommiwe have done separate definitions for state machines, as pure data too. signals to the state machine have (input) schemas, guards, actions and arbitrary other control data to them and the uis are derived from the combination of fme + schema definitions: form can be derived from schemas, available actions (and popovers why they are not available) from guards and the available actions in from the fme step definitions. lot’s of ui-stuff like extra re-frame action definitions, button texts and model guidees in the process definitions too.#2020-10-2119:39ikitommimost real uis need more that “automatically derived from schema”, but, it seems good for many cases like erp’ish internal stuff. Finland is the land of forms, great to have some extra leverage to build those ;)#2020-10-2119:41ikitommigreat talk by @U050CBXUZ & @U0DJK1VH6 on building declarative uis with clj: https://www.youtube.com/watch?v=IekPZpfbdaI#2020-10-2210:39borkdude@ikitommi I was noticing a regex/parse function in one of the examples. Why is there a special regex/parse function: shouldn't all schema's support parse - not only regex schemas?
And maybe parse is just a special case of a transformer?#2020-10-2210:41borkdudeNote: this question is coming from a relatively inexperienced user of malli#2020-10-2210:51ikitommi@borkdude good question! it’s the internal api, not yet integrated into malli.core. after it’s integrated, there will ne a`-destructure` or -parse method in Schema. My first guess was that there is no need to have a reverse function for that in Schema (like spec has conform and unform), but not 100% sure.#2020-10-2210:54ikitommioh, haven’t pushed the current draft into github. will have time tomorrow for this, will push the stuff out.#2020-10-2509:49borkdudehttps://www.reddit.com/r/Clojure/comments/jhq899/did_anyone_migrate_a_nontrivial_project_from_spec/#2020-10-2510:14jeroenvandijkI did, yes it is more involved 🙂#2020-10-2510:17jeroenvandijkTo be more specific, this project is a (conceptual) port from a clojure.spec project https://github.com/jeroenvandijk/aws.cloudformation.malli#2020-10-2510:18jeroenvandijkI was generating the clojure.spec definitions from the AWS Cloudformation spec. This generation part was much easier with Malli#2020-10-2510:19jeroenvandijkThere are enough differences that I cannot consider it as a 1 on 1 port#2020-10-2510:40borkdudeFeel free to respond on Reddit as this was not my own question, just forwarded it here#2020-10-2510:51jeroenvandijkyeah thanks. I’m not using reddit. I hope the reddit user finds it here 😅#2020-10-2516:28borkdudeIs there any guidance on how to write your own transformer?#2020-10-2516:35ikitommi@borkdude there is no guide yet, but lot of tests and some samples in the README.#2020-10-2516:36ikitommihappy to help if you need any.#2020-10-2516:38borkdude@ikitommi I think I asked this before, but I can't find any docs on this, nor can I find the conversation on Zulip. So here it goes:
(def Schema
[:map [:x int?]])
(defrecord Wrapper [obj loc])
parsed
;;=> {#script.Wrapper{:obj :x, :loc {:row 1, :col 2, :end-row 1, :end-col 4}} #script.Wrapper{:obj 1, :loc {:row 1, :col 5, :end-row 1, :end-col 6}}}
(defn wrapper-transformer []
(mt/transformer
{:name :wrapper
:default-decoder :obj
:default-encoder (fn [obj]
(->Wrapper obj nil))}))
(prn (m/decode Schema parsed wrapper-transformer)) ;;=> nil ... ???
#2020-10-2516:40borkdudeso I want to validate/transform the wrapped value according to the schema#2020-10-2516:41borkdudeany value that is not Wrapped should just be transformed as is#2020-10-2516:41borkdudebut a value that is wrapped should be unpacked using :obj#2020-10-2516:42borkdudeand I want to give an error message based on :loc#2020-10-2516:44ikitommioh, i recall. need to dig in my histories too if there was/is a good answer.#2020-10-2516:45borkdudeI tried this:
(defn unwrap [x]
(if (instance? Wrapper x)
(:obj x)
x))
(defn wrapper-transformer []
(mt/transformer
{:name :wrapper
:default-decoder unwrap
:default-encoder (fn [obj]
(->Wrapper obj nil))}))
but this just returns the entire object itself#2020-10-2516:47borkdudeit seems it doesn't handle the value recursively#2020-10-2516:50ikitommithe problem is that the decoder is first given a map, it calls :obj on it, which return nil.#2020-10-2516:51ikitommiI would use clojure.walk/pre-walk for first sweep of decoding.#2020-10-2516:52ikitommithe decoding here walks:
1. the :map
2. the keys (`:x`)#2020-10-2516:52ikitommiunless the Wrappedis already decoded on 1, there is no :x and the decoding stops.#2020-10-2516:53ikitommiso, with malli, you would need to decode all the keys & values on :map step also. doable, but extra noise 😞#2020-10-2516:55ikitommi(ns user
(:require [malli.core :as m]
[malli.transform :as mt]))
(def Schema
[:map [:x int?]])
(defrecord Wrapper [obj loc])
(def parsed {(map->Wrapper {:obj :x, :loc {:row 1, :col 2, :end-row 1, :end-col 4}})
(map->Wrapper {:obj 1, :loc {:row 1, :col 5, :end-row 1, :end-col 6}})})
(defn unwrap [x]
(if (instance? Wrapper x) (:obj x) x))
(defn wrapper-transformer []
(mt/transformer
{:name :wrapper
:decoders {:map (fn [x] (reduce-kv (fn [acc k v] (assoc acc (unwrap k) (unwrap v))) {} x))}
:default-decoder unwrap
:default-encoder (fn [obj] (->Wrapper obj nil))}))
(m/decode Schema parsed wrapper-transformer) ;;=> nil ... ???
; => {:x 1}#2020-10-2516:55ikitommibut, this is much simpler:
(clojure.walk/prewalk unwrap parsed)
; => {:x 1}#2020-10-2516:57borkdudeYes, but how do I get validation errors if I first call prewalk on this thing?#2020-10-2516:58borkdudebased on :loc#2020-10-2516:58ikitommiI’ll check more of this later.#2020-10-2517:02borkdudeThanks. Me too, cooking dinner :)#2020-10-2517:07borkdudeHmm, in spec I would maybe have to spec a key as ::wrapped-int and then coerce it after it was checked?#2020-10-2517:13borkdudewhich is not ideal#2020-10-2517:25ikitommican the wrapped by anywhere? Wrapped map/vector/set? All values are wrapped (any nested edn value)? Or just keys and values in the map?#2020-10-2517:26ikitommiI have an idea, but would like to understand the case first.#2020-10-2517:54borkdude@ikitommi The use case is preserving location information for non-iobjs and using that for error messages while validating malli schemas#2020-10-2517:54borkdudee.g. you want to validate an EDN file and you get an error: this should be an int, on line 5, row 12#2020-10-2518:16borkdude@ikitommi This is the complete code:
deps.edn:
{:deps {metosin/malli {:mvn/version "0.2.1"}
borkdude/edamame {:git/url ""
:sha "ba93fcfca1a0fff1f68d5137b98606b82797a17a"}}}
(ns script
(:require [edamame.core :as e]
[malli.core :as m]
[malli.transform :as mt]))
(def Schema
[:map [:x int?]])
(defrecord Wrapper [obj loc])
(defn iobj? [x]
(instance? clojure.lang.IObj x))
(def parsed
(e/parse-string "{:x 1}"
{:postprocess
(fn [{:keys [:obj :loc]}]
(if (iobj? obj)
(vary-meta obj merge loc)
(->Wrapper obj loc)))}))
(defn unwrap [x]
(if (instance? Wrapper x)
(:obj x)
x))
(defn wrapper-transformer []
(mt/transformer
{:name :wrapper
:default-decoder unwrap
:default-encoder (fn [obj]
(->Wrapper obj nil))}))
;; (prn parsed)
(prn (m/decode Schema parsed wrapper-transformer))#2020-10-2519:35borkdude@ikitommi Found the conversation here: https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/malli/near/206235546#2020-10-2519:52borkdude@ikitommi So the way it can work is:
(defn fail! [schema {:keys [:obj :loc]}]
(throw (ex-info (str obj " did not satisfy " schema
" [at " (str (:col loc)":"(:row loc)) "]") {})))
(def <42 [:and 'int? [:< 42]])
(defn lift-non-iobj-schema [schema]
[:map {:encode/success :obj,
:encode/failure (partial fail! schema)} [:obj schema]])
(def Schema [:map [:a (lift-non-iobj-schema <42)]])
(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))#2020-10-2519:52borkdudeE.g. this will print:
(prn (parse-validate-and-transform "{:a 42}"))#2020-10-2519:52borkdudeSyntax error (ExceptionInfo) compiling at (script.clj:32:1).
42 did not satisfy [:and int? [:< 42]] [at 5:1]
#2020-10-2519:53borkdudeThe downsize of this is that I have to wrap schemas myself in case I want to check something non-iobj-ish (strings, keywords, numbers)#2020-10-2519:57borkdudeAnd this doesn't seem to work for keywords for example:
(def <42 [:and 'int? [:< 42]])
(defn lift-non-iobj-schema [schema]
[:map {:encode/success :obj,
:encode/failure (partial fail! schema)}
[:obj schema]])
(def Schema [:map [(lift-non-iobj-schema :a) (lift-non-iobj-schema <42)]])#2020-10-2519:58borkdudeanyway, maybe this is a too niche use case#2020-10-2520:03ikitommihave an idea, need 30min to test & play#2020-10-2520:05borkdudetake yer time#2020-10-2520:06borkdudeif this can be fixed, potentially it would also work for rewrite-cljc structures: normal malli specs, but they run over the rewrite-cljc nodes using a simple function that looks at the actual value#2020-10-2520:07borkdudeit's basically a projection from wrapped thing to the essential value while the wrapped thing has more info to produce useful output in case of failure to parse/validate#2020-10-2610:16ikitommi@jarvinenemil the huge schema creation perf issue: culprit is satisfies?, which can be removed using a special extending protocols to protocols pattern. With quick test it’s 22sec -> 2sec. With some tuning, 0.8sec. Fix is on the backlog.#2020-10-2610:17dangercoderawesome @ikitommi, good find!#2020-10-2621:26mruzekwHi all, what are the options currently to use Malli like you can with clojure.spec and ghostwheel or orchestra? So far I've found teknql/aave, but I wondered if there were others#2020-10-2623:04rschmukler@mruzekw I think aave is the only one right now. It's definitely been on my backburner though so happy to accept PRs if you feel like hacking on it. Otherwise I wouldn't be surprised if ikitommi ends up releasing his own thing. Happy to update aave to 0.2.0 if that's what is keeping you back from it. (No pressure to use it either!)#2020-10-2716:43lmergenjust a shout-out that i'm biting the bullet and migrating our project with 4k+ LoC full of specs to malli and i'm totally loving it -- the fact that it's just plain symbols + data makes thing so much easier to reason about. thanks a lot for this monumental effort of making malli work!#2020-10-2716:48lmergenjust a sanity check, but it seems to me that using "plain data" for all the schemas (i.e. not wrapping them in m/schema), and then at a later point use a compile-time m/validator and m/explainer is a decent approach, right ?
i've just whipped up this macro to perform assertions, which allocates the explainer at compile time, but does the actual validations at runtime
(defmacro assert
[s v]
(when *assert*
(let [explainer# `(m/explainer ~s)]
`(when-let [ed# (~explainer# ~v)]
(throw (ex-info "Validation failed"
(merge {::v ~v}
(me/humanize ed#))))))))
#2022-09-2300:52marciolHey, @lmergen thanks for this snippet.
I wonder if this can be part of the default malli distribution @U055NJ5CC#2022-09-2313:24ikitommiI think that m/assert would be good, but it should work in all situations, so I believe the schema creation should happen at runtime.#2020-10-2716:51borkdude@lmergen I'm curious about your experience report. There is a thread on Reddit about exactly this. Feel free to comment there if you want.#2020-10-2716:51borkdude(https://www.reddit.com/r/Clojure/comments/jhq899/did_anyone_migrate_a_nontrivial_project_from_spec/)#2020-10-2716:52rschmukler@lmergen that's exactly what I do. Most (all?) of the malli public API will coerce to a compiled schema, which lets you keep your schema definitions nice and focused. Then compile-time m/validator and m/explainer. Also there are the short hands of m/explain (and m/validate) which create a temporary explainer for you. This isn't performant, but, for use-cases like your macro, it saves you having to create it yourself.#2020-10-2716:53rschmuklerI suppose a slight difference is that your macro will create the explainer at compile time, so up to you if you want that difference!#2020-10-2716:53lmergen@borkdude i started that thread 🙂#2020-10-2716:54borkdudeaaah! :)#2020-10-2716:57lmergenbut yes so far so good -- of course there's a "second system" benefit here, because i have a much better understanding of the model and taking the opportunity to refactor a few things. one of the big changes is that rather than one humongous specs.cljc, i'm now creating many more namespaces, and each of the malli schemas live within those namespaces. the reason for this is that spec works with (namespaced) keywords, so you can get away with putting them in a single namespace, where malli seems to map more natural if you split things up in different namespaces.#2020-10-2717:04rschmukler@lmergen that was exactly my experience as well! One benefit that you get is if you make use of def or defn to return those data schemas, you get the added benefit of having the compiler help check that those things still exist. In spec you can delete / mistype a keyword and get bit, but here, you can get the compiler (or clj-kondo) to help you#2020-10-2810:12borkdude@ikitommi I think I would have a real good use of sequence schemas. Spec is painful to work for command line usage with due to macros, malli is a pleasure because it's data and also compatible with Graal.
Context: https://gist.github.com/borkdude/a391146ad81a06c28fb97ccdc1f64d44#2020-10-2810:41jeroenvandijkI was thinking about this too. To find duplicated or similar code etc#2020-10-2810:13borkdudeImagine that the user can specify a schema on the command line to search for code patterns.#2020-10-3000:05eraadHi! I’m using Malli schemas to generate a Swagger spec using Reitit (ie. “issue_date”). These schemas have encoding fns so “issue_date” is coerced to :document/issue-date so it can be saved into Datomic.
I want to confirm if the Reitit coercers do validation after coercion right? Because I’m not getting it to validate as “issue_date” but :document/issue-date#2020-10-3000:07eraadIt is what I can see from the code and from the behaviour. If I want to keep the json-like schemas for Swagger generation AND validate using those, I need to do coercion after all that manually right?#2020-10-3000:08eraadOr of course write my own Malli coercer that always decodes.#2020-10-3000:12eraadI would appreciate any pointers on this#2020-10-3016:01ikitommi@lmergen @rschmukler should there be m/assert?#2020-11-0108:59lmergeni think there should be -- i have two functions myself actually, assert and have, where have is inspired by ptaoussanis' truss -- https://github.com/ptaoussanis/truss
effectively, have does the assertion + also returns the input value if no failure was detected. this allows for a pattern such as
(let [foo (m/have foo-schema (get x :foo))]
...)
it's very effective.
i would be happy to send a PR for the assert and/or have implementations i have right now (which seem to be doing the right thing)#2020-10-3016:02ikitommi@eraad coercion is decode + validate in reitit-malli#2020-10-3016:03ikitommifor responses, I recall it's validate + encode#2020-10-3016:04ikitommihave had a super busy two weeks, should have few full days of malli after next wed.#2020-10-3016:21eraadThanks @ikitommi got you#2020-10-3016:30ikitommithe m/defn, currently have a separate malli.schema ns, and few helper ns's under it. Will have the 1:1 Plumatic Schema m/defn for Malli. Once done, will do the real full malli->clj-kondo integration, that can be used with both malli-defn and aave, and a standard fdef -kinda way to annotate existing functions like in spec.#2020-10-3016:31borkdude@ikitommi exciting!#2020-10-3016:32ikitommimight be worth trying to write spec->malli conversion, so we could reuse all the existing fdefs out there. Also, zillion little things to do :)#2020-10-3019:55rschmukler@ikitommi once we have that I may take a crack at some nrepl middleware and cider integration. Right now you can spec generate and see spec definitions in documentation lookup in emacs. Only thing I miss from spec.#2020-10-3019:56rschmuklerRe: m/assert - why not? I think it'll be attractive for some of those migrating from spec#2020-10-3100:24hoppyjust for funzies, this warning is coming out of shadow-cljs in a node-js release build. I'll be happy to chase it if somebody wants me to. Oh, and p.s., malli running on AM335x embedded device, in case somebody is keeping score.#2020-11-0115:26ikitommi@hoppy https://github.com/metosin/malli/issues/87 help welcome!#2020-11-0118:29lmergenis it correct that (at least currently), transforming enums from json does not yet correctly transform them to keywords if necessary ?#2020-11-0118:54borkdude@lmergen Should enums always be keywords?#2020-11-0118:57lmergenwell I wouldn’t say always — but at least I would expect it to convert it if that leads to a valid enum value#2020-11-0118:58borkduderight#2020-11-0119:16ikitommi@lmergen right. There is no type inferrer with enums. They could be anything [:enum 1 "2" :3 '4]. If you know the type, you can hint it: [:and keyword? [:enum :a :b :c]] and they transform correctly#2020-11-0119:16lmergenoh that works well enough for now #2020-11-0119:19ikitommithe generated JSON Schema with :and is bit off, it creates :allOf out of that. Should merge the results instead if there is just one district :type found.#2020-11-0119:20ikitommi(correct, but not practical)#2020-11-0211:29lmergenis this a valid instant? it can be represented by Instant, but it appears to not be parsable by malli.transform:
(pr-str (mt/-string->date "0000-12-31T23:59:59.999Z"))
=> "0000-12-31T23:59:59.999Z"
#2020-11-0211:31lmergen1. Caused by java.time.DateTimeException
Invalid value for YearOfEra (valid values 1 -
999999999/1000000000): 0
apparently that's the issue#2020-11-0211:33lmergenseems like a inconsistency to me to what Instant can represent and DateTime can parse#2020-11-0220:40ikitommi@lmergen inst? should be companioned with set of date-types, the issue is here: https://github.com/metosin/malli/issues/49. That said, we just merged an enhancement to inst? transformation, happy to take another one, if there is something still missing. See https://github.com/metosin/malli/pull/280#2020-11-0220:41lmergenI’ll take a look, thanks! This is a corner case anyway#2020-11-0220:42lmergenlooks like this may indeed fix my issue as it’s the same instant -> date -> instant round trip, I’ll give it a try #2020-11-0311:32Markus StrHi, I'm getting started with malli and this migh be a noob question, but why is this an invalid schema error? (I took the source of string? in cljs and changed it to str2? Weirdly string? is valid and the alias isn't
(defn ^boolean str2?
"Returns true if x is a JavaScript string."
[x]
(goog/isString x))
(def Test
[:map
[:url str2?]
[:det map?]
]
)
(-> Test
(m/explain {:url "sa" :det {:a 5}})
(me/humanize)
)#2020-11-0311:34Markus StrWhy can't I define a boolean predicate function there?#2020-11-0311:41borkdude@US22FPMPX Any reason you're not using string?#2020-11-0311:44Lucy Wang@US22FPMPX try [:fn str2]#2020-11-0311:51ikitommithe default shemas are listed here: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1223-L1269#2020-11-0311:53borkdudeFWIW string? on CLJS is:
(defn ^boolean string?
"Returns true if x is a JavaScript string."
[x]
(goog/isString x))
#2020-11-0311:53Markus Strthanks for the quick repsonse
@U04V15CAJ, str2? was just for figuring out if my function was the issue#2020-11-0311:54borkdudeah ok#2020-11-0311:54Markus Str@UP90Q48J3 thanks that works actually, but error messages are not defined then#2020-11-0311:54Markus Strexample:
(defn twitter-url? [s] (re-find #"http[s]?://twitter.*\d+" s))
(def Test
[:map
[:url [:fn twitter-url?]]
[:det map?]
]
)
(-> Test
(m/explain {:url "" :det {:a 5}})
(me/humanize)
)#2020-11-0311:55Markus Str(-> Test
(m/explain {:url "" :det {:a 5}})
(me/humanize)
)#2020-11-0311:55Markus Str{:url ["unknown error"]} then#2020-11-0311:55borkdude@US22FPMPX Have you tried https://github.com/metosin/malli#custom-error-messages?#2020-11-0311:56borkdudeMaybe malli expects the function to return a boolean instead of nil? don't know#2020-11-0311:58Markus StrThanks for the pointer#2020-11-0311:59ikitomminils should be ok, also regexs:
[:map
[:x #"http[s]?://twitter.*\d+"]
[:y [:fn {:error/message "invalid"} (constantly false)]]#2020-11-0312:08Markus StrAh interesting, so regex work, but I can't have something like:
[:x #(re-find #"http[s]?://twitter.*\d+" %)]
Probably have to add to the default schema registry then, to have it work naturally like, [:x twitter-url?] I guess?#2020-11-0312:10Markus StrI'm going to read the docs for the third time; seems my mental model is not there yet!#2020-11-0312:13Markus Str(defn twitter-url? [s] (re-find #"http[s]?://twitter.*\d+" s))
(def -twitter-url? [:fn {:error/message "invalid twitter url"} twitter-url?])
(def Test
[:map
[:url -twitter-url?]
[:det map?]
]
)
(-> Test
(m/explain {:url "" :det {:a 5}})
(me/humanize)
)
So this is how I could do it; only backdraw is that I have two functions now; one for normal use, the other specifically for validation#2020-11-0312:16ikitommicould make the predicate schemas implement IFn, so you could:
(def twitter-url?
(m/schema
[:re {:error/message "invalid twitter url"}
#"http[s]?://twitter.*\d+"]))
(twitter-url? "")
; => true
(m/validat twitter-url? "")
; => true#2020-11-0312:28Markus StrLooks promising. For me personally (and I might be totally off) it could be nice for malli to also accept predicate functions directly inside the spec-vector
(defn twitter-url? [s] (re-find #"http[s]?://twitter.*\d+" s))
(def Test
[:map
[:url twitter-url?]
]
)
For now I'll read up on the registry and how to add it there#2020-11-0312:34borkdudeWhat's the reason it requires [:fn ..] right now @U055NJ5CC?#2020-11-0316:21ikitommicould add an option to allow that shortcut (as there is regexs), but in short: it’s too easy to write schemas which do not serialize.#2020-11-0316:22ikitommibut if one is not looking for serialization, it would be easier for sure.#2020-11-0316:22borkdudegood point. maybe for some people serializing a schema is not important?#2020-11-0316:22borkdudejinx#2020-11-0316:23ikitommineed to revisit these for 1.0.0. but will add optional support for plain functions.#2020-11-0316:24ikitommialso, the default registry being immutable. it’s currently easy to use immutable registries, but hard to get a global mutable registry. both could be easy.#2020-11-0316:26borkdudeDo you think malli will be able to do something like grasp?#2020-11-0316:27borkdudeas in, support the things that spec does to match s-expressions#2020-11-0316:27ikitommiI believe so, as soon as there are the sequence schemas.#2020-11-0412:13Lucy Wang> good point. maybe for some people serializing a schema is not important?
to be frank I think quite some (if not most) people don't need that ...#2020-11-0313:13lmergenis there something off with generators and the [:and] clause ?
(def schema [:and map?
[:map
[:foo string?]]])
(mg/generate schema)
throws an error ("Couldn't satisfy such-that predicate after 100 tries.")
this, however, works:
(def schema [:and
[:map
[:foo string?]]
map?])
(mg/generate schema)
my hunch is that it's unable to "deduce" that it should first try to generate the :map , and starts by generating a completely random map which of course never qualifies the [:map [:foo string?]] schema?#2020-11-0313:17ikitommiyes, the order matters#2020-11-0313:31lmergenstill seems like there's something off with :and :thinking_face:
(def base [:map
[:foo [:enum :a :b]]])
(def x [:and base
[:multi {:dispatch :foo}
[:a [:map [:i string?]]]
[:b [:map [:j pos-int?]]]]])
seems like mg/generate is unable to generate values for this schema as well#2020-11-0314:25ikitommi@lmergen
(defmethod -schema-generator :and [schema options] (gen/such-that (m/validator schema options) (-> schema (m/children options) first (generator options)) 100))
#2020-11-0314:26ikitommi^:-- the first one is used for the generator.#2020-11-0314:26lmergenaha!#2020-11-0314:26lmergenbut that means there isn't really a way for me to work around it except writing a custom generator, right ?#2020-11-0314:28lmergen(which is what i just did)#2020-11-0314:29ikitommiyou could merge the base schemas, but merge might not work with :multi. but, there are options, like:
(mg/generate
[:and
[:multi {:dispatch :foo}
[:a [:merge ::base [:map [:i string?]]]]
[:b [:merge ::base [:map [:j pos-int?]]]]]]
{:registry (merge
(m/default-schemas)
(mu/schemas)
{::base [:map [:foo [:enum :a :b]]]})})#2020-11-0314:30lmergeni see#2020-11-0314:31ikitommimerge doesn’t work with :multi currently, this would be best way to do it (if it worked):
(mu/merge
[:map [:foo [:enum :a :b]]]
[:multi {:dispatch :foo}
[:a [:map [:i string?]]]
[:b [:map [:j pos-int?]]]])
;[:multi {:dispatch :foo}
; [:a [:map [:i string?]]]
; [:b [:map [:j pos-int?]]]]#2020-11-0314:32lmergeni just wrote a helper function which just does this:
(defn merged
"Given a list of generators, returns a generator that applies `merge`
to all the generator results."
[& gens]
(gen/fmap (fn [args]
(reduce merge {} args))
(apply gen/tuple gens)))
(def schema [:and {:gen/gen (merged (mg/geneator a) (mg/generator b)} a b])
#2020-11-0314:32ikitommi:multi should do type-inferring, so that all map-like :multis act like a map.#2020-11-0314:32lmergenyes#2020-11-0314:32lmergenthat's what i was going to say#2020-11-0314:32lmergen:multi is pretty much always a map anyway#2020-11-0314:34ikitommiunless it’s a tuple / sequence:
(m/validate
[:multi {:dispatch 'first}
[:sized [:tuple keyword? [:map [:size int?]]]]
[:human [:tuple keyword? [:map [:name string?] [:address [:map [:country keyword?]]]]]]]
[:human {:name "seppo", :address {:country :sweden}}])
; true#2020-11-0314:34lmergenright#2020-11-0314:35ikitommibut, it’s easy to figure out if it looks like a map and could be merged.#2020-11-0319:02Maciej FalskiHi, I’ve just started with malli, and I’m spiking converting our spec definitions to malli-based. Among others, we’ve defined specs with generators and json decoding for java-time (aka java8) types , like instant, duration , and interval. It’s easy with spec-tools. Now I’m trying to do the same with malli and this is what I came up with:
(def Instant
(m/-simple-schema
{:pred t/instant?
:type-properties {:error/message "should be an instant"
:decode/json t/instant
:encode/json str
:gen/gen (gen/fmap (fn [[d h m s]] (t/plus (t/instant) (t/days d) (t/hours h) (t/minutes m) (t/seconds s)))
(gen/tuple gen/int gen/int gen/int gen/int))}}))
Does it makes sense? Would there be a better way? I’m aware of this https://github.com/metosin/malli/issues/49, but it’s still open.#2020-11-0406:13ikitommi@maciej.falski looks good! few small things:
• if you want to support also string->edn & edn->string, you should add :decode/string and :encode/string keys. Needed if you use the dates in header/path/yaml/query-params
• you could add :type key, .e.g. 'Instant so that the schema forms render correctly
• with :type, you could also add the new schmas into registry and they are available for de/serialization too.#2020-11-0406:16ikitommioh, and 'going to talk about mallin in London Clojurians in 8.12. https://www.meetup.com/London-Clojurians/events/274367598/#2020-11-0508:51ikitommiback to function schemas. basics kinda work without varargs, as it requires sequence schemas. todo: emit clj-kondo annotations for real.#2020-11-0508:52ikitommi#2020-11-0508:54ikitommialso todo: pretty errors, e.g. https://github.com/metosin/virhe#2020-11-0906:47ikitommi@borkdude tested the malli + edamame with location preserving metadata. Few notes:
• current impl of transform is not aware of the path where the transformation happens. If it did, the solution would be almost trivial
• good thing is that we have other tools to go around the limitation: we can walk the schema and inject :in to all subschema properties and use :compile in a transformer to read this info at decoder creation time and to collect all value path -> loc into a request-scoped atom as we decode the Wrapped values
• transformers compose, so we can also do string->edn etc in a single sweep
• for the transformed value, we call explain and in case of errors, attach the locs from the atom, as explain errors know already both schema and value paths
• solution has lot of boilerplate, and found a :set explain bug (wrote https://github.com/metosin/malli/issues/294 ), with that, this could even work 😎
• https://gist.github.com/ikitommi/e3229a0bcef532d1fa032321713227d3#2020-11-0906:48ikitommigood thing about using a lookup-table is that it doesn’t need to throw and reports all errors.#2020-11-0906:51ikitommialso, if edamame could accumulate and provide the full path to a given element in :postprocess, it could be used here too, e.g. extra key + value in Wrapped like :in [:tags "address"].#2020-11-0906:53ikitommi(def Address
[:map
[:id string?]
[:tags [:set keyword?]]
[:address
[:map
[:street string?]
[:city string?]
[:zip int?]
[:lonlat [:tuple double? double?]]]]])
;; string->edn, no coercion
(let [coerce (coercer Address)]
(coerce (slurp "schema.edn")))
;{:schema [:map
; [:id string?]
; [:tags [:set keyword?]]
; [:address
; [:map
; [:street string?]
; [:city string?]
; [:zip int?]
; [:lonlat [:tuple double? double?]]]]],
; :value {:id "Lillan",
; :tags #{":hotel" :coffee :artesan},
; :address {:lonlat [61.4858322 23.7854658],
; :city "Tampere",
; :street "Ahlmanintie 29",
; :zip "33100"}},
; :errors (#Error{:path [:tags 0],
; :in [:tags 0], ;; <--- the set value paths are incorrect #294
; :schema keyword?,
; :value ":hotel",
; :loc {:row 2, :col 10, :end-row 2, :end-col 18}}
; #Error{:path [:address :zip],
; :in [:address :zip],
; :schema int?,
; :value "33100",
; :loc {:row 5, :col 17, :end-row 5, :end-col 24}})
; :string "{:id \"Lillan\"
; :tags #{:artesan :coffee \":hotel\"}
; :address {:street \"Ahlmanintie 29\"
; :city \"Tampere\"
; :zip \"33100\"
; :lonlat [61.4858322, 23.7854658]}}
; "}
;; string->edn, with malli string-coercion
(let [coerce (coercer Address (mt/string-transformer))]
(coerce (slurp "schema.edn")))
; => nil#2020-11-0914:02ikitommi@borkdude #294 is fixed in master and the edamame-walking works now and is bit simpler:
1. parse with edamame
2. prewalk twice to get both the original EDN and the path-vec -> loc lookup table
3. glue things together
4. kudos to @nilern for a working walker
5. https://gist.github.com/ikitommi/e3229a0bcef532d1fa032321713227d3#2020-11-0914:03ikitommi#2020-11-0914:05ikitommiit automatically binds a transformer named :edamame, so you can add custom decoding hints to schemas:
[:string {:decode/edamame str/upper-case}]
… and if sci is enabled, the schemas can be read from files too.#2020-11-0918:53lmergeni have a particularly complex schema where the initialization of e.g. (m/validator) or (m/transformer) is fairly slow -- about 300ms. one way to deal with this would be to cache these -- is my understanding correct that using a registry will effectively do this ? or will i need to write my own caching layer on top of it ?#2020-11-0918:54lmergenor are registries just a very simple way of organizing stuff, without any pre-parsing going on ?#2020-11-0919:09ikitommi@lmergen the schema creation will get a 10x boost soon, the slow part being m/schema. If you add Schema instances into registry, it happends just once. Or you can just use a var:
(def Address (m/schema [:map [:street :string]]))
#2020-11-0919:10ikitommi… but, for super fast validation, you should just create the m/validator once and reuse that. it returns a pure and optimized function.#2020-11-0919:12lmergenright, I think I’ll just go for that last option. fairly often various of these validators are used in hot code paths, so I’ll probably write something to cache validators instead. #2020-11-0919:13lmergenbut if I wrap things in an m/schema call, it’ll already do a lot of preprocessing, right?#2020-11-0919:15ikitommiright. I’ll run some flamegraphs. just a sec.#2020-11-0919:24ikitommi(time
(prof/profile
(dotimes [_ 50000]
(m/validate [:map [:street :string]] {:street "hämeenkatu"}))))
;; "Elapsed time: 10472.153783 msecs"
(let [schema (m/schema [:map [:street :string]])]
(time
(prof/profile
(dotimes [_ 500000]
(m/validate schema {:street "hämeenkatu"})))))
;; "Elapsed time: 231.093848 msecs"
(let [validate (m/validator [:map [:street :string]])]
(time
(prof/profile
(dotimes [_ 500000]
(validate {:street "hämeenkatu"})))))
;; "Elapsed time: 59.743646 msecs"#2020-11-1318:02hoynkMay I ask what profile lib you are using?#2020-11-1318:21ikitommimostly https://github.com/clojure-goes-fast/clj-async-profiler g https://github.com/hugoduncan/criterium#2020-11-1318:50hoynkthx#2020-11-0919:25lmergenright, this makes a lot of sense. #2020-11-0919:25ikitommi#2020-11-0919:26ikitommim/schema uses satisfies? which has a perf issue, most of the time spent there.#2020-11-0919:30ikitommior was it two orders of magnitude? satisfies? seems to take at least 95% of the time.#2020-11-0919:30ikitommi#2020-11-0919:37lmergenok, this is very helpful#2020-11-0919:41lmergenI really do find the validators to be significantly faster than spec validate — it’s about 3x faster for my fairly insane schema (the same that takes 300ms to parse).
better yet, and this was unexpected: the generators are also much faster. I’m not 100% certain yet because whether this is because maybe Malli takes some shortcuts, but i seem to be able to avoid a few annoying gen/such-that? generators with Malli that causes a very large speed up. #2020-11-1009:21ikitommithis is interesting. tried to avoid such-that?, e.g. setting min & max when known, but have not tested against spec gen.#2020-11-0921:43ikitomminew flames with cache#2020-11-0921:45ikitommi10472ms => 568ms (18x faster)
#2020-11-1007:18lmergen@ikitommi for what it's worth, i still got a huge performance increase by actually caching the validators as well.
(crit/with-progress-reporting
(crit/quick-bench (m/validate schema value)))
;; => Execution time mean : 297.880813 ms
(def schema' (m/schema schema))
(crit/with-progress-reporting
(crit/quick-bench (m/validate schema' value)))
;; => Execution time mean : 533.885193 µs
(def validator (m/validator schema))
(crit/with-progress-reporting
(crit/quick-bench (validator value)))
;; => Execution time mean : 1.830348 µs
so it looks like about a 500x improvement by caching schemas, and then another 300x improvement by caching the validators#2020-11-1007:21lmergeni suspect in your specific benchmark, the schema is fairly simple so then a larger share of the benchmark is actually about performing the validation#2020-11-1009:18ikitommi@lmergen there was a cljs-issue, just merged the cached satisfies. could you retry with the latest master?#2020-11-1009:19lmergensure!#2020-11-1009:19lmergen1 minute#2020-11-1009:20ikitommithere is still a lot of room for improvement for maps (`-parse-entries` is really slow) and for handling property-based registries. I would guess can make schema creation 2-5 times faster. But then again, after malli is used to validate schema properties & children, it will slow things down again.#2020-11-1009:29lmergen(crit/with-progress-reporting
(crit/quick-bench (m/validate schema value)))
;; before: => Execution time mean : 297.880813 ms
;; after: => Execution time mean : 12.194964 ms
(def schema' (m/schema schema))
(crit/with-progress-reporting
(crit/quick-bench (m/validate schema' value)))
;; before: => Execution time mean : 533.885193 µs
;; after: => Execution time mean : 517.890217 µs
(def validator (m/validator schema))
(crit/with-progress-reporting
(crit/quick-bench (validator value)))
;; before: => Execution time mean : 1.830348 µs
;; after: => Execution time mean : 1.952607 µs
so while m/validate got ~ 20x faster, caching the actual validator is still much, much faster#2020-11-1009:40lmergeni'm caching the explainers in my own defn macro, but it requires quite a bit of macro magic to make this work, so i was looking for a more generic way to make this happen -- possibly some kind of registry#2020-11-1009:43ikitommiwhat should be in the registry? validator + explainer + generator + decoder(s) + encoder(s) + …?#2020-11-1009:44lmergenif possible, i'd say all of them yes#2020-11-1009:44ikitommiin reitit, there is a Coercion protocol to cache things relevant there: https://github.com/metosin/reitit/blob/master/modules/reitit-malli/src/reitit/coercion/malli.cljc#2020-11-1009:45lmergenright, so then you lazily cache things#2020-11-1009:45lmergenwhich would be the best middle-ground#2020-11-1009:47ikitommione would be to add a wrapper Schema impl, that is returned from registry instead of the real one. And that impl would have a cache -> first call to -validate would store the validator.#2020-11-1009:47lmergenoh i see#2020-11-1009:47ikitommicould be just an option to the registry to return caching proxys instead of normal ones…#2020-11-1009:47lmergenyes#2020-11-1009:47lmergenthis would be very effective#2020-11-1009:48lmergeni'll experiment with this approach#2020-11-1009:50ikitommi… actually, just a new option key that m/schema uses would do fine (to wrap the returned thing if the option is present)#2020-11-1010:02lmergenso then you cache it inside the actual schema, rather than a wrapper around it ?#2020-11-1010:06ikitommiI would wrap it outside, e.g. the return value wrapped#2020-11-1010:06lmergenah, right -- and the option to m/schema would then tell it whether to return the wrapped schema or the "regular" schema#2020-11-1010:09ikitommiyes. Or there could be a memoized-schema etc. as a separate fn? (-> :string m/schema m/memoized)#2020-11-1010:09lmergenyes#2020-11-1010:09lmergenwell that's a detail#2020-11-1010:09lmergenlet me experiment with creating that memoized / cached schema in the first place#2020-11-1016:52lmergen(crit/with-progress-reporting
(crit/quick-bench (m/validate schema value)))
;; => Execution time mean : 11.782073 ms
(def schema' (memoized-schema (m/schema schema)))
(crit/with-progress-reporting
(crit/quick-bench (m/validate schema' value)))
;; => Execution time mean : 2.095245 µs
@ikitommi conceptually it seems to be working like a charm#2020-11-1016:55lmergenexactly which schema am i supposed to wrap here -- it's just the regular malli.core/Schema, right ? the into-schema is meant more for building a hierarchy of parent/child schemas ?#2020-11-1016:57ikitommiyes Schema. IntoSchema is the factory-protocol for creating a Schema out of the Schema AST, each Schema is responsible for it’s own props & children.#2020-11-1016:57lmergenright#2020-11-1016:57lmergenawesome#2020-11-1016:57lmergeni'll send a PR once i have all the functions working#2020-11-1115:54Maciej FalskiHey, I think I found a bug while trying to get malli coercion working with reitit. Basically it comes to this:
; based on example
(def Over6
(m/-simple-schema
{:type :user/over6
:pred #(and (int? %) (> % 6))
:type-properties {:error/message "should be over 6"
:decode/string mt/-string->long
:json-schema/type "integer"
:json-schema/format "int64"
:json-schema/minimum 6
:gen/gen (gen/large-integer* {:min 7})}}))
(mu/closed-schema Over6)
=> #'user/Over6
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :user/over6}
I can raise an issue in GH if you can confirm it.#2020-11-1214:24Rodrigo A. RoveriFolks, I have a begginer's question.
I would like to test Malli, so I created a new lein project and included malli in the dependencies.
I was able to run lein repl, but was unable to follow the examples on the doc page. (see image below)
Am I missing something? Some pre-requisite? I tested on these two versions "0.0.1-SNAPSHOT", "0.2.1".#2020-11-1214:24Rodrigo A. Roveri#2020-11-1214:34borkdudeWhat does the exception preceding your REPL expressions say?#2020-11-1214:53Rodrigo A. Roveri#2020-11-1214:53Rodrigo A. Roveri@borkdude#2020-11-1214:54borkdudeHow are you starting this repl? what is in your project.clj?#2020-11-1214:56Rodrigo A. RoveriOh oh. I think maybe I know the answer. My project is also named "malli":man-facepalming:#2020-11-1214:57Rodrigo A. Roveri#inception#2020-11-1214:58Rodrigo A. RoveriThank you very much. Keep up the good work. I will run some tests. The project seems really interesting!#2020-11-1217:13ikitommi@maciej.falski oh, that’s bad, didn’t realize that unregistered Schemas can’t be walked with the current impl of m/schema-walker, as it tries to recreate the schmas using m/type, which in this case is :user/over6 and not found in the registry.#2020-11-1217:13ikitommiMight need to add a -copy method into Schema protocol.#2020-11-1217:14ikitommigood thing is, that a per-schema -copy is most likely near to no-op, which makes Schema walking easily order(s) of magnitude faster.#2020-11-1217:37Maciej FalskiNot sure if it’s about the registry, ie:
(mr/set-default-registry!
{:string (m/-string-schema)
:over6 Over6})
=> #object[malli.registry$simple_registry$reify__4072 0x1f74f3fd "#2020-11-1217:42ikitommi@maciej.falski you should register it with :user/over6 so it matches the Schema definition#2020-11-1417:26ikitommiSchema knows now the IntoSchema that creates it. Fixes the issue @maciej.falski and makes schema copying (= all utils) faster.#2020-11-1419:47ikitommitwo ways to create reusable rules for mutually exclusive keys for maps:
1. just data:
(defn exclusive-keys [keys]
[:fn {:error/message (str "the following keys are mutually exclusive: " (str/join ", " keys))}
(fn [m] (not (every? (partial contains? m) keys)))])
(-> [:and
[:map
[:foo {:optional true} :int]
[:blabla {:optional true} :int]
[:hah {:optional true} :int]
[:bar {:optional true} :int]]
(exclusive-keys [:hah :bar])
(exclusive-keys [:hah :foo])]
(m/explain {:foo 1 :hah 2})
(me/humanize))
; => #:malli{:error ["the following keys are mutually exclusive: :hah, :foo"]}
2. new Schema impl (with pretty form):
(def Exclusive
(m/-simple-schema
(fn [_ [keys]]
{:type 'Exclusive
:min 1
:max 1
:pred (fn [m] (not (every? (partial contains? m) keys)))
:type-properties {:error/message (str "the following keys are mutually exclusive: " (str/join ", " keys))}})))
(-> [:and
[:map
[:foo {:optional true} :int]
[:blabla {:optional true} :int]
[:hah {:optional true} :int]
[:bar {:optional true} :int]]
[Exclusive #{:hah :bar}]
[Exclusive #{:hah :foo}]]
(m/explain {:foo 1 :hah 2})
(me/humanize))
; => #:malli{:error ["the following keys are mutually exclusive: :hah, :foo"]}#2020-11-1419:49ikitommino macros = awesome#2020-11-1419:52ikitommithe latter is the not-well-documented reagent-style thing, any IntoSchema can be used in first position in Schema AST and -simple-schema allows one to read the properties & children at creation time and return a Schema instance based on those.#2020-11-1419:52ikitommiit’s bit like derived types schemas?#2020-11-1509:30ikitommiwrote an issue for discussing about auto-updating with container schemas: https://github.com/metosin/malli/issues/304. Comments most welcome.#2020-11-1511:37ikitommifunction schemas, comments welcome: https://github.com/metosin/malli/issues/125#issuecomment-727555388#2020-11-1511:40borkdudeI think you are ignoring the fact that different arities can have different return types/schemas#2020-11-1511:41borkdudeI think clojure.spec is also making this too difficult. In clj-kondo I chose to define the spec per arity#2020-11-1511:41borkdudeIt is more verbose, but at least easy to verify#2020-11-1511:42ikitommigood point, the schema also forces the return to be the same for all arities.#2020-11-1511:42borkdudereally?#2020-11-1511:43ikitommiI guess there are lot of examples in the core where the different arities return different things?#2020-11-1511:43ikitommiyes.#2020-11-1511:44borkdudeconfirmed:
> - The output schema always goes on the fn name, not the arg vector. This
means that all arities must share the same output schema.#2020-11-1511:45borkdudewell, clojure.core/map, filter, etc, is an example that have different return types for different arities. I kinda wish that the transducer arity was just a different version like clojure.core/mapping but that ship has sailed. I often make mistakes with this#2020-11-1511:45borkdudelook at the spec that will result from this:
https://github.com/borkdude/speculative/blob/master/src/speculative/core.cljc#L297-L302#2020-11-1511:46borkdudeI think this is not ergonomic at all. spec has to do backtracking etc, to match the right arity.#2020-11-1511:46borkdudeimo the fn spec should match the structure of the defn args+bodies#2020-11-1511:49ikitommigood point. what would be a good malli definition for this:
(defn fun
([x] x)
([x y] [x (* x x)]))
#2020-11-1511:50ikitommiclj-kondo has a good syntax for the different arities, use that’ish?#2020-11-1511:50ikitommisomething like:
(m/=> fun {:arities {1 {:output int?
:input [:tuple int?]}
2 {:output [:tuple int? pos-int?]
:input [:tuple int? int?]}}})#2020-11-1511:51borkdudeThat's how I do it in clj-kondo yes.#2020-11-1511:52borkdudeNot sure if that's the best, but I optimized for matching speed, so clj-kondo can find the right arg types fast#2020-11-1511:53borkdudeas for defn syntax, maybe:
(defn fun
([x :- int?] :- int?
x)
([x :- int? y :- string?] :- [:tuple string? int?]
[y (* x x)]))
#2020-11-1511:53borkdudeso the return type directly after the arg vec#2020-11-1511:54ikitommi👍#2020-11-1511:57ikitommithe extracted schemas could be in the compact/short :=> format, which is always 1-arity:
(defn fun1 [x] (* x x))
;; short
(m/=> fun1 [:=> int? [:tuple pos-int?]])
(defn fun
([x] (fun x x))
([x y] [x (* x x)]))
;; short
(m/=> fun [:or
[:=> int? [:tuple int?]]
[:=> [:tuple int? pos-int?] [:tuple int?]]])
#2020-11-1511:59ikitommithe long versions:
(defn fun1 [x] (* x x))
;; long
(m/=> fun1 {:arities {1 {:input int?
:output [:tuple pos-int?]}}})
(defn fun
([x] (fun x x))
([x y] [x (* x x)]))
;; long
(m/=> fun {:arities {1 {:output int?
:input [:tuple int?]}
2 {:output [:tuple int? pos-int?]
:input [:tuple int? int?]}}})
#2020-11-1511:59borkdudecould work yes#2020-11-1511:59borkdudedoesn't fn spec hinge on sequence specs which are not exposed yet?#2020-11-1512:01ikitommiyes, those are needed for varags. we just had an internal tech-talk on friday, did a plan how to get the sequence schemas & schema parsing out. takes few days to make that good.#2020-11-1512:01ikitommifirst demo of the function schemas will be with non-vargargs. enough to get feedback & start with the clj-kondo integration.#2020-11-1516:09ikitommi(require '[malli.schema :as ms])
(ms/defn ^:always-validate fun :- [:tuple int? pos-int?]
"returns a tuple of a number and it's value squared"
([x :- int?] :- any? ;; arity-level override
(fun x x))
([x :- int?, y :- int?] ;; uses the default return
[x (* x x)]))
(clojure.repl/doc fun)
; -------------------------
; demo/fun
; ([x] [x y])
;
; [:-> [:tuple int?] any?]
; [:-> [:tuple int? int?] [:tuple int? pos-int?]]
;
; returns a tuple of a number and it's value squared
#2020-11-1516:16ikitommifull meta:
(ms/defn square :- pos-int?
[x :- int?]
(* x x))
(meta #'square)
;{:schema [:or [:-> [:tuple int?] pos-int?]],
; :ns #object[clojure.lang.Namespace 0x3c5f3ba8 "demo"],
; :name square,
; :file "/Users/tommi/projects/metosin/malli/src/malli/schema.cljc",
; :column 1,
; :raw-arglists ([x :- int?]),
; :line 64,
; :arglists ([x]),
; :doc "\n[:or\n [:-> [:tuple int?] pos-int?]]"}#2020-11-1520:28ikitommi#2020-11-1606:04ikitommifirst draft of m/=>, to annotate existing functions. now - just with the compact/mallli notation, could support the map-version too. Kinda like it as it’s as long as defn, to the two can be aligned.#2020-11-1606:05ikitommi… supports multi-artities with different return for each.#2020-11-1610:50ordnungswidrigmagic#2020-11-1708:45ikitommithere are some gaps in the malli->clj-kondo integration, any insights welcome: https://github.com/metosin/malli/blob/cde74871ee5edc1525344f7a6e62d54fb1b00f5b/src/malli/clj_kondo.cljc. marked with ;;??#2020-11-1708:45ikitommicurrently:
(require '[malli.clj-kondo :as mc])
(mc/transform
[:map
[:id string?]
[:tags {:optional true} [:set keyword?]]
[:address
[:map
[:street string?]
[:city string?]
[:zip {:optional true} int?]
[:lonlat [:tuple double? double?]]]]])
;{:op :keys
; :opt {:tags :set}
; :req {:id :string
; :address {:op :keys
; :opt {:zip :int}
; :req {:street :string
; :city :string
; :lonlat [:double :double]}}}}#2020-11-1708:51ikitommibtw, would be great if clj-kondo supported :min and :max for numbers & collections. nice demo about “more than types”, something like:
(ms/defn times :- :int
"times"
[x :- [:int {:min 3}]]
(* x x))
(times 2)
;; ^:--- clj-kondo error of "number should be at least 3"#2020-11-1709:25borkdudeDon't know if this a common enough problem in function calls. I haven't encountered this much I think. If there's a domain specific need to check this, one can also write a hook#2020-11-1709:32ikitommiok. just wondering as clj-kondo already supports map-op-syntax, so if :int could be defined as {:op :int}, it would allow adding easily new keys that would be easy to check, e.g. malli could emit {:opt :int, :min 3} from the example. Not sure how useful that would be, but:
• number min & max might be easy to implement on clj-kondo side?
• would be a great demo, as you can’t easily present these with simple typed langs liike Java#2020-11-1709:32ikitommibut, I don’t need this for real, just thinking aloud about a kick-ass demo 🙂#2020-11-1709:33borkdudeit has :pos-int and :nat-int which map to the clojure predicates. it basically only has that which exists as predicates in clojure 1.9+#2020-11-1709:36borkdude@ikitommi Feel free to try out things with the clj-kondo type system. You can build a custom clj-kondo for your demo possibly#2020-11-1709:36borkdudeit's all in clj-kondo.impl.types#2020-11-1709:36ikitommithanks, might give it a shot.#2020-11-1709:42borkdude@ikitommi Ah, I can see why your approach might be problematic:
https://github.com/borkdude/clj-kondo/blob/master/src/clj_kondo/impl/types.clj#L157-L163
So the literal value is mapped to a type tag. And then during checking that is resolved very cheaply, not executing any predicates, but just checking a keyword identity or a super simple graph, e.g. a pos-int is also an int#2020-11-1709:44borkdudeso the literal value is erased as it were#2020-11-1718:47HankstenbergHi guys, just getting my feet wet with malli. Quick question: when I try to transform a map to JSON-Schema, all the map's keywords are still Clojure keywords and thus invalid syntax in JSON-Schema. Sure, I could post-process it, but is this on purpose? What's the best way to get proper JSON-Schema?#2020-11-1719:03ikitommi@marcus.poparcus I think it would be better to emit strings directly. PR Welcome.#2020-11-1820:34HankstenbergThe simplest way to do it would probably be to wrap the output in clojure.data.json/write-str, looks pretty ugly of course and becomes useless in clojureland. I guess it really depends on what consumes the output afterwards. Maybe that should be handled case by case. How about an additional function "transform-json-str"? Comes with the new dependency of clojure.data.json, though. Looking at the source code of clojure.data.json it's not trivial to generate valid JSON manually.#2020-11-2001:36fmnHi guys, is there any way to get all of the values defined in :enum ? I'm trying to do some testing and it would be great if I could just iterate through it rather than hard-coded it. I guess I could just (next (malli.core/form my-enums)) . Just trying to find a more consistent way since this breaks if I add options to it.#2020-11-2004:46ikitommi@funyako.funyao156 try (m/children MyEnun)#2020-11-2215:58ikitommihi all. comments on the function schema syntax? some alternatives:
;;
;; many ways to present function schemas
;; - 2 ints to int
;; - int to int
;; - no args to int
;; - no args to irrelevant
;;
;; 1: current
[:=> [:tuple int? int?] int?]
[:=> [:tuple int?] int?]
[:=> [:tuple] int?]
[:=> [:tuple]]
;; 2: shortcut
[:=> [int? int?] int?]
[:=> [int?] int?]
[:=> [] int?]
[:=> []]
;; 3: separator
[:fn int? int? :=> int?]
[:fn int? :=> int?]
[:fn :=> int?]
[:fn :=>]
#2020-11-2303:58rutledgepaulvI personally do not mind the more verbose version. I am wary of adding too many syntactic shortcuts since sometimes they begin to collide in unfortunate ways that make it more difficult to write correct tooling. But maybe you already have easy ways to quickly convert them into a normal form?#2020-11-2306:51ikitommino tools to convert, totally agree that there should not be extra syntaxes in malli core.#2020-11-2307:08Martín VarelaI like the 3rd version best, seems easier to read than the other ones.#2020-11-2307:17ikitommi@U95NTJT4H that is basically Ghostwheel (spec) / Aave (malli) do:
(>defn bad-return-val
[x y]
[int? int? => string?]
(+ x y))
#2020-11-2307:19ikitommione more:
;; 4: via properties
[:=> {:input [:tuple int? int?]
:outut int?}]
[:=> {:input [:tuple int?]
:output int?}]
[:=> {:output int?}]
:=>#2020-11-2307:20Martín VarelaI haven't used those, but it does seem more readable than the alternatives (I guess this is a very subjective thing, anyway). I guess I'm also with @U5RCSJ6BB in not minding a bit more verbosity#2020-11-2307:20Martín Varelaterse is really cool, until it isn't... 🙂#2020-11-2307:23ikitommidata-specs for spec-tools for a great idea for really simple things. But as soon as one needed something non-trivial, it became a burden. Same with spec2, s/select syntax is awesome, until you need something inlined specs in it.#2020-11-2307:23Martín VarelaBTW, is the idea to provide the full power of malli for these? If I think of my use cases, the most likely ones would probably be rather complicated crap data (nested, with constraints, etc..), so it'd be cool to be able to use a registry or similar#2020-11-2307:24Martín Varelathey may become rather unreadable very fast, if inlined as such#2020-11-2307:25ikitommi:=> is just a normal Schema, so generators, validators etc. work normally#2020-11-2307:25Martín Varelaok#2020-11-2307:31Martín VarelaNot entirely unrelated to this, the other day, while trying to figure out that reitit issue, I thought the problem was that reitit wasn't picking up the registry, so I ended up writing a bit of meander term rewriting code that allows you to define schemas based on other schemas (where you'd normally use a registry), and compile those to malli "primitives". So you get both the concision and explicitness, in a way.#2020-11-2311:47rutledgepaulvSuppose another question is should a "irrelevant" return be a special case or should you just use a 'any? schema#2020-11-2216:03ikitommiI think as Aave already provides the short variant, so the official (malli-like) syntax doesn’t have to be that terse, just formal. But the 1 is still.. ugly.#2020-11-2306:53ikitommiplaying with :=> generators. also good peek at the syntax options (using children vs properties):
(def Age
[:int {:min 18, :max 100}])
(def User
[:map
[:name string?]
[:age Age]
[:skills [:set [:enum "clj" "cljs" "rust" "go" "cobol"]]]])
;; option1: childs
(def create-user
[:=> {:gen/=> '(fn [[age] user] (assoc user :age age))}
[:tuple Age] User])
;; option2: properties
(def create-user
[:=> {:gen/=> '(fn [[age] user] (assoc user :age age))
:input [:tuple Age]
:output User}])
(mg/generate User {:size 10})
; => {:name "Szf5sN", :age 21, :skills #{"cobol" "rust" "cljs" "clj" "go"}}
(def create-user-gen (mg/generate create-user {:size 10}))
(create-user-gen 46)
; => {:name "Itu96", :age 46, :skills #{"cobol" "rust" "cljs" "clj" "go"}}#2020-11-2321:31ikitommialso, validating functions against mallidefinitions:
(def Age
[:int {:min 18, :max 100}])
(def User
[:map
[:name string?]
[:age Age]
[:skills [:set [:enum "clj" "cljs" "rust" "go" "cobol"]]]])
(mg/generate User {:size 10})
; => {:name "Szf5sN", :age 21, :skills #{"cobol" "rust" "cljs" "clj" "go"}}
(def create-user [:=> [:tuple Age] User])
(m/validate
create-user
(fn [age] {:name "elephant", :age age, :skills #{"cobol"}})
{::m/=>validator mg/=>validator})
; => true
;; invalid return
(m/validate
create-user
(fn [age] {:name "elephant", :age age, :skills #{"PERL"}})
{::m/=>validator mg/=>validator})
; => false
;; invalid arity
(m/validate
create-user
(fn [age _extra] {:name "elephant", :age age, :skills #{"cobol"}})
{::m/=>validator mg/=>validator})
; => false#2020-11-2415:18heliosIt's not clear how i could say that i want a non-empty map in my schema, that also works with generators 🙂#2020-11-2415:27helios(mg/generate [:and
[:fn not-empty]
[:map-of
[:string {:gen/gen gen/string-ascii}]
[:string {:gen/gen gen/string-ascii}]]])
this ofc works for validation but not with generators#2020-11-2415:53ikitommi@U0AD3JSHL try reversing that, :map-of as first child. :and generates based on first, and narrows with the rest.#2020-11-2415:54heliosuh, nice one 🙂#2020-11-2421:19Hankstenbergmalli is absolutely awesome, thanks a lot guys! 👍#2020-11-2421:29ordnungswidrigIs there a “canonical representation” for malli? I guess it’s the “vector” syntax as opposed to the “map” syntax, right?#2020-11-2505:45steveb8n@ikitommi is ::keys in this line a bug? https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L337#2020-11-2505:48ikitommi@U0510KXTU not a bug, it uses qualified keys for configuration: https://github.com/metosin/malli/blob/master/test/malli/transform_test.cljc#L207#2020-11-2505:49steveb8nok. TIL a new destructing syntax 🙂#2020-11-2505:50ikitommiyes, a handy way to do that. Also,
{:keys [::json-vectors]} would work.#2020-11-2505:53ikitommi@ordnungswidrig right, it the vector syntax for now.#2020-11-2511:37Elsois there something like a best-effort implementation of a json-schema to malli converter out there?#2020-11-2515:19ikitommi@d.eltzner012 don't think so, there is an old PR, but lot's of todos and last update from jul.#2020-11-2515:19ikitommihttps://github.com/metosin/malli/pull/211#2020-11-2515:22rutledgepaulvI have a half-baked implementation of this for converting kubernetes swagger into malli. I'm using malli to write a clojure kubernetes client with client side validation of operations (similar to aws-api). It handles recursive schemas at least but some of the kubernetes swagger is on the janky side so i'm sure it's not properly general. The whole library is still early stages. https://github.com/RutledgePaulV/kube-api/blob/master/kube-api-core/src/kube_api/core/swagger/malli.clj#2020-11-2618:33ikitommi👍#2020-11-3008:24Elso👍 very cool still, I'll have a look at it#2020-11-2611:02HankstenbergThe more I look at it the more I want to use malli schemas as a single source of truth for everything. Are there any resources about generating ui models from malli schemas? I think the idea to integrate it with Domino was floating around once, but the current version doesn't mention malli. How else could it be done? My go-to framework for UI stuff is usually re-frame. Malli could be integrated so that re-frame's db is generated from it. And to validate state changes, of course. It could also be bound to visual representations of actual components I guess. Would be grateful for any reference to existing work in this area.#2020-11-2614:32rutledgepaulvI have played with this a bit. I eventually came to a realization that I should first transform malli into an intermediate representation prior to generating the UI state because there are additional things you'll want to know about each node, like whether the field is "dirty" and has been "visited", etc. So in other words.. first I would go through the work of figuring out a sensible IR and generating forms from that, and then work backwards and generate that initial IR from malli.#2020-11-2614:35rutledgepaulvThat said, my initial attempt of using malli directly (and still struggling with how to do the reagent cursors) is here: https://rutledgepaulv.github.io/ui-kit/#!/ui_kit.forms https://github.com/RutledgePaulV/ui-kit/blob/master/src/cljs/ui_kit/visitors.cljs#2020-11-2618:53ikitommiHave seen multiple prototypes. Did also a demo into on of our projects. There is malli.util/subschemas helper which takes a Schema and returns an ordered list of all the schema paths and Schemas in those, handling also collection schemas. One place to start. Also, one can post-walk the Schemas and wrap all schmas into something that can be used to strore the form state (dirty, pristine, etc). hope this helps.#2020-11-2618:59ikitommi#2020-11-3006:53HankstenbergThanks a lot for the input! I'm still trying to wrap my head around everything. I think Datascript (re-posh)+datasync are the right tools to connect front-end state and back-end state, I think I will try to find a way to make this stack (malli-)schema-driven.#2020-11-2618:53ikitommiHave seen multiple prototypes. Did also a demo into on of our projects. There is malli.util/subschemas helper which takes a Schema and returns an ordered list of all the schema paths and Schemas in those, handling also collection schemas. One place to start. Also, one can post-walk the Schemas and wrap all schmas into something that can be used to strore the form state (dirty, pristine, etc). hope this helps.#2020-11-2619:03rutledgepaulvWonder if there's an opportunity to improve error messages here:#2020-11-2619:03rutledgepaulv(def stream
(logs client
{:operation "readCoreV1NamespacedPodLog"}
{}))
Execution error (ExceptionInfo) at kube-api.utils/validation-error (utils.clj:87).
Invalid request.
{:path-params ["missing required key"]}
(def stream
(logs client
{:operation "readCoreV1NamespacedPodLog"}
{:path-params {}}))
Execution error (ExceptionInfo) at kube-api.utils/validation-error (utils.clj:87).
Invalid request.
{:path-params
{:name ["missing required key"], :namespace ["missing required key"]}}
#2020-11-2619:04rutledgepaulvif there's a missing key, insert the key with an empty value and run validation on that empty value too so you can report not only the missing collection but also what needs to be in the collection?#2020-11-2619:04rutledgepaulvprobably breaks down in some edge cases.. thinking#2020-11-2619:06ikitommiyou could run default-value-transformer to add empty maps first.#2020-11-2619:08ikitommi(m/decode
[:map
[:path-params [:map
[:name :string]
[:description :string]]]]
nil
(mt/default-value-transformer
{:defaults {:map (constantly {})
:string (constantly "")}}))
; => {:path-params {:name "", :description ""}}#2020-11-2619:14rutledgepaulvthanks! yes, that's working for me#2020-11-2914:05joshkhnoob question, but how can i validate/generate non-blank strings? with spec i would use (complement clojure.string/blank?) .
(m/validate [:map [:name [:and string? (complement clojure.string/blank?)]]] {:name ""})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema #object[clojure.core$complement$fn__5669 0x394ebe0b "
#2020-11-2914:39joshkhthis validates, but fails on generation
(m/validate [:map [:name [:fn (complement clojure.string/blank?)]]] {:name "test"})#2020-11-2915:13eval2020@U0GC1C09L the README has [:string {:min 1}] as example.#2020-11-2915:18joshkhnot sure how i missed that. thanks eval2020.#2020-11-2915:35ikitommiI think there should be a built-in complement.#2020-11-2915:35ikitommidoesn’t exist, but should.#2022-07-0208:52Eric Dvorsak[:string {:min 1}] isn't the same, the string could still be blank, eg " "#2020-11-2915:36ikitommimerged into master the current status of function schemas, alpha, feedback welcome! the guide is here: https://github.com/metosin/malli#function-schemas#2020-11-2915:36ikitommiand the PR https://github.com/metosin/malli/pull/306#2020-11-2915:37ikitommialso has now the full clj-kondo integration !!#2020-11-2915:38ikitommiwill make a separate PR of the optional plumatic-style helpers.#2020-11-2915:40ikitommithere are now var instrumentation helpers atm, only shipped use case is the clj-kondo type checker integration. Any ideas what would be a good api to instrument functions in dev to run the input & output checks?#2020-11-2915:40ikitommiany ideas about clojure.test helpers for malli?#2020-11-2915:41ikitommidev-time tooling has not been the goal for malli (as it’s the primary goal for spec already), but happy to add things to that direction too.#2020-11-2915:42ikitommiwith spec, have used https://github.com/jeaye/orchestra, I guess like everyone else 🙂#2020-11-3007:22HankstenbergQuick question: Is is possible to combine malli and spec-tools in order to get both, the JSON schema and the API schema? Do you plan to extend spec-tools or malli? Actually it has to be malli right? Yesterday I realized how awesome programatically updating malli specs is. You can have a complex user schema with an ":active" boolean in in and all you need to do to specify an inactive user is to say (def inactive-user (mu/assoc user :active [:enum false]). It's like a language-agnostic type that would be perfectly exportable to e.g. openapi3 and JSON form to make use of the tools available for these standards. (trying to find good reasons to introduce clojure with malli as a way to generate openapi/json schema in our company that uses them).#2020-12-0518:14ikitommiSlow answer 😉 : I’m not sure I understand the question, but you can generate JSON Schema from malli. Glad you like it.#2020-12-0707:07HankstenbergYea, I know and I'm loving it. I'd like to experiment a bit with API-Driven development at our company. OpenApi is nice, but who wants to write JSON/YAML by hand when you can write edn instead and programmatically create JSON Schema and OpenAPI-Specs from that. The JSON Schema part is covered nicely by malli, which is wonderful, because we can utilize the nice ecosystem that uses JSON Schema already AND have the power of property-based testing.
But what about speccing the endpoints? There is spec-tools, but spec-tools uses clojure.spec instead of malli. I imagine that I could e.g. specify an "Order" schema and a "Processed Order" schema in order to specify an entity and an entity in a certain state (with "processed" set to "true" and a "processed-on" field set to some date and so on.
With the ability to even spec functions with malli, we have all the building blocks to extend malli's capabilities to also specify the actual API:
(def =>process-oder [:=> [:tuple entities/order events/process-order] entities/processed-order])
Now all that's missing is to map this to e.g. a POST request and generate a complete openapi-spec from it (schema+end points), so that e.g. a REST-Client can be generated from this code.
At least that's how I would imagine it. Maybe I'm on the wrong track here, though.#2020-12-0707:36ikitommihave you looked at reitit-malli? here’s an example: https://github.com/metosin/reitit/tree/master/examples/ring-malli-swagger#2020-12-0708:59HankstenbergThat's the best possible reply. Yes, there already is something and btw it's even better than what you had in mind. Thanks a bunch! 👍#2020-12-0115:05mynomotoAbout the clj-kondo integration, is malli annotations compatible with the clj-kondo ones? As in if something is typed string on malli and is passed to a core function that only accepts numbers, will it show as a warning?#2020-12-0115:06borkdude@mynomoto this doesn't work automatically, you should execute some code to make malli spit out type information to a clj-kondo config directory and you should opt in to this config dir via :config-paths#2020-12-0115:09mynomotoI'm assuming that the developer did the correct project configuration.
The question is more about if a malli string is a clj-kondo string when doing the type inference.#2020-12-0115:10borkdude@mynomoto it should:
https://github.com/metosin/malli/blob/23772defaa6223529edc316fd1ba8fb0c87ae3f0/src/malli/clj_kondo.cljc#L23#2020-12-0115:13mynomotoThis is great, thanks! I was wondering if a project to add malli type annotations to core functions would be useful for type inference but I'm glad that it is not necessary.#2020-12-0413:39Adrian SmithHey I'm trying out the clj-kondo integration, I've used the latest sha for malli in my deps.edn and have a working clj-kondo installation, I can see some clj-kondo configuration has been emitted by malli but the plus example doesn't appear to be erroring, is this ready for looking at?#2020-12-0515:33rutledgepaulvis there a recommended way to write a function that will dispatch to an implementation according to which of several schemas match my input arguments? I can construct my own of course that just does a linear search, and i could create a multi schema to compose them together (though i would need to use a compound discriminator for my case). I guess what i'm wondering is if there's a way to do pattern matching using malli schemas, but preferably in an open way like multimethods so i can just accrete new cases instead of modifying a case expression.#2020-12-0515:35rutledgepaulvmaybe i'm just rubbing up against open arbitrary predicate dispatch at that point. hm#2020-12-0515:37rutledgepaulvperhaps i could add an optional attribute on each schema (in my case they are open maps so this is fine) and supply a :default attribute. then i can run a default transformer on my inbound value to inject a "type" and from there just use regular multimethods to dispatch on that attribute.#2020-12-0515:57rutledgepaulvyeah that works. sorta rough and not sure the macro is a good idea but in case anyone is curious:#2020-12-0515:57rutledgepaulv(defn dispatchable [dispatch-key schema]
(mu/update-properties
schema
(fn [props]
(let [dispatch-decoder (fn [value] (with-meta value {:dispatch dispatch-key}))]
(assoc-in props [:decode/dispatch] {:leave dispatch-decoder})))))
(defn dispatch-key [schema value]
(some-> (m/decode schema value (mt/transformer {:name :dispatch})) meta :dispatch))
(defmacro defdispatchable [symbol & body]
`(def ~symbol (dispatchable ~(name symbol) (do
#2020-12-0515:57rutledgepaulv(handle-auth {:tokenFile "st"})
token file!
=> nil
(handle-auth {:client-key-data "st"
:client-certificate-data "sfd"})
client key!
#2020-12-0516:20rutledgepaulvActually, i think this is much better:#2020-12-0516:20rutledgepaulv(def schema-resolver-transformer
(mt/transformer
{:default-decoder
{:compile (fn [schema _]
(fn [value]
(if (instance? IObj value)
(vary-meta value assoc :resolved schema)
value)))}}))
(defn resolve-schema [schema value]
(some-> (m/decode schema value schema-resolver-transformer) (meta) :resolved))
(def token-file-auth
[:map {:dispatch :token-file}
[:tokenFile :string]])
(def client-key-auth
[:map {:dispatch :client-key}
[:client-key-data :string]
[:client-certificate-data :string]])
(def combined
[:or token-file-auth client-key-auth])
(defn dispatch-fn [context]
(let [schema (resolve-schema combined context)]
(:dispatch (m/properties schema))))
(defmulti handle-auth #'dispatch-fn)
(defmethod handle-auth :token-file [context]
(println "token file!"))
(defmethod handle-auth :client-key [context]
(println "client key!"))#2020-12-0516:30rutledgepaulvTommi, let me know if there's a better way to do this ^. Providing a way to resolve the concrete schema that is used for each node in a tree of data allows me to put useful data into the schema properties and recover that data for a given value so that i can act on it (like dispatching to different functions in my code based on which schema matched). Here i am attaching the schema to the data as metadata (which makes sense but will not work for values that don't implement IObj)#2020-12-0518:08ikitommi@sfyire @borkdude I think one needs to add extra entry into :config-paths into .clj-kondo/config.edn. I have the following:
✗ cat .clj-kondo/config.edn
{:config-paths ["configs/malli"]}
forgot all about that, will add it to README if that is the required glue here.#2020-12-0518:12ikitommihttps://github.com/metosin/malli/commit/da45eceaea3e5b34426027bf7836e7cbe841cc97#2020-12-0518:08ikitommiwould be great if that was not needed and it would just work.#2020-12-0518:09ikitommie.g. with that, clj-kondo pics up the emitted file, looking something like this:
✗ cat .clj-kondo/configs/malli/config.edn
{:lint-as #:malli.schema{defn schema.core/defn}
:linters {:type-mismatch {:namespaces {user {square {:arities {1 {:args [:int], :ret :nat-int}}}
plus {:arities {1 {:args [:int], :ret :int}
2 {:args [:int :int], :ret :int}}}}}}}}#2020-12-0518:31ikitommi@rutledgepaulv your second options looks interesting, tagging values. I most likely would do it like that. One problem I see with this approach is that the meta-data can get out-of-sync: if your say (dissoc data :tokenFile) , it still is tagged as :token-file, but it not anymore valid against that schema. Might not be a problem if you just use it once, but relying of :resolved meta in general has not guarantees.
The Schema parsing is currently WIP (first release coming soon), with that, you will have a variant to :or (and :alt), which will have named branches, like spec does, so you can say ~about this:
(def token-file-auth
[:map
[:tokenFile :string]])
(def client-key-auth
[:map
[:client-key-data :string]
[:client-certificate-data :string]])
;; using named :or variant
(def combined
[:or*
[:token-file token-file-auth]
[:client-key client-key-auth]])
(m/parse combined {:token-file "kikka"})
; => #Branch{:key :token-file
:value {:token-file "kikka"}}
#2020-12-0518:34rutledgepaulvthanks! the schema parsing stuff looks like that would address my issue too#2020-12-0518:34ikitommiyou could also use m/encode and return a tuple yourself, with both the schema/name and the value.#2020-12-0518:35rutledgepaulvyeah. in my case i only need to know the concrete version of the top level schema so tuples would be fine#2020-12-1008:23Kari Marttilahttps://www.youtube.com/watch?v=bQDkuF6-py4, Tommi’s presentation in London Clojurians. This is a great presentation. The presentation does a great job explaining the differences between different data validation libraries and the various ways you can utilize the malli library for data validation.#2020-12-1008:26Kari MarttilaI found particularly interesting the beginning of the presentation. Tommi does a good job visualizing how Plumatic Schema, Spec and Malli specify an example data.#2020-12-1008:44borkdudeAre the slides available publically?#2020-12-1008:47dharriganThat was a great talk btw. I use malli (with Reitit) a lot!#2020-12-1009:03ikitommithanks!#2020-12-1009:03ikitommislides are here: https://www2.slideshare.net/metosin/designing-with-malli#2020-12-1401:11rutledgepaulvI am a little confused by map entries in malli. I see that i can use malli.core/entries to get k/v tuples where the v is a reified :malli.core/val schema that contains the entry's properties, but i sort of expected the map entry itself to also be a reified schema node that I could interrogate properties from. For now I am listing all entries and seeking to find the one that matches my key since malli.util/get returns the value not wrapped in a :malli.core/val and so is missing entry properties#2020-12-1401:11rutledgepaulvIt might be nice if malli.util contained a "find" like "clojure.core/find" for getting a map entry given a key in constant time#2020-12-1401:12rutledgepaulvam i missing something? i'm sure map entries are some of the clunkier things to deal with#2020-12-1401:13rutledgepaulvMaybe what i am missing is that my property is really about the "value" and not about the map entry and so i put it in the wrong place.#2020-12-1911:13raymcdermottHey @ikitommi we recently had @dominicm on defn and he took issue with my pronunciation of Malli. But just listened to your excellent talk at re:Clojure and your pronunciation matches my own. He thinks you’re wrong 😂#2020-12-1911:14raymcdermottIs it your Tampere accent?#2020-12-1911:15dominicmOh no, ousted! I don't even remember disagreeing. Maybe I just like contradicting you @raymcdermott#2020-12-1911:15raymcdermottOh no you don’t #2020-12-1911:15raymcdermott[ sorry it’s Pantomime season ]#2020-12-1911:16raymcdermottAnyway we’ll let you get on with more interesting topics :)#2020-12-1911:22borkdudeBtw, I enjoyed the podcast, but I'm now also aware of how dominicm manipulated me and Alex to do stuff for him!#2020-12-2114:33ikitommi@raymcdermott listened that defn, good stuff. Finnish is simple, just pronounce the way you write it, right? 😉#2020-12-2114:35ikitommibut, reitit is pronounced ray-tit.#2020-12-2117:26dominicmCan confirm, Ray is a tit ;) @raymcdermott#2020-12-2114:36ikitommi@rutledgepaulv the entry handling is bit too hard atm, but not sure what woud be a simpler way.#2020-12-2114:36ikitommithere is a way to ask for the entry instead of the value, just a sec.#2020-12-2115:31ikitommioh, undocumeted, untested and had a bug, but maybe something like this?
(mu/find
[:map [:b {:optional true} int?]]
:b)
; => [:b {:optional true} int?]
(-> [:map [:a [:map [:b {:optional true} int?]]]]
(mu/get :a)
(mu/find :b))
; => [:b {:optional true} int?]#2020-12-2115:33ikitommiwhere mu/find is just sugar on top of the hidden feature of “give me the entry tuple” of:
(-> [:map [:a [:map [:b {:optional true} int?]]]]
(mu/get-in [:a [::m/find :b]]))
; => [:b {:optional true} int?]#2020-12-2116:12ikitommihttps://github.com/metosin/malli/pull/322#2020-12-2116:13ikitommibtw, those interested in the sequence schemas, here’s the PR (draft): https://github.com/metosin/malli/pull/317#2020-12-2116:17ikitommi(m/explain
[:+ [:cat*
[:prop string?]
[:val [:alt* [:s string?] [:b boolean?]]]]]
["-server" :foo "-verbose" true "-user" "joe"])
=>
{:schema [:+ [:cat* [:prop string?] [:val [:alt* [:s string?] [:b boolean?]]]]],
:value ["-server" :foo "-verbose" true "-user" "joe"],
:errors (#Error{:path [0 :val :s], :in [1], :schema string?, :value :foo}
#Error{:path [0 :val :b], :in [1], :schema boolean?, :value :foo})}#2020-12-2222:41elaroussHi, Is it possible to write a schema for this vector:
[{:type :work
:value "
the vector should only contain those 2 items, where the type is kind of static, the value can change.
i thought about using this schema:
[:vector [:map [:type keyword?] [:value string?]]]
but it’s not enough for my use case, because i want when i inspect the schema to know how many items i can have in the vector, and their types (the :type )
is this possible?
Thank you for the great lib 🙏#2020-12-2223:14alpox@U6AE62UCT I can think of two ways:
(def data-schema [:map
[:type [:enum :work :personal]]
[:value string?]])
for the maps and then
(def schema [:vector {:min 2 :max 2} data-schema])
or maybe
(def schema [:tuple data-schema data-schema])
for putting together the schema for the vector#2020-12-2307:12elaroussI was thinking about something like
[:vector
[:map [:type :work] [:value string?]]
[:map [:type :personal] [:value string?]]]
but it didn’t work
Thank you @U6JS7B99S i’ll probably go with your first suggestion#2020-12-2308:08ikitommiif the :type defines the shape of the data, you can try :multi:
(require '[malli.generator :as mg])
(mg/sample
[:vector {:min 2, :max 2}
[:multi {:dispatch :type}
[:work [:map [:type [:= :work]] [:value string?]]]
[:personal [:map [:type [:= :personal]] [:value string?]]]]])
(mg/sample
[:tuple {:registry
{::element
[:multi {:dispatch :type}
[:work [:map [:type [:= :work]] [:value string?]]]
[:personal [:map [:type [:= :personal]] [:value string?]]]]}}
::element ::element])#2020-12-2308:09ikitommiboth emit results like:
([{:type :personal, :value ""} {:type :work, :value ""}]
[{:type :work, :value "l"} {:type :personal, :value ""}]
[{:type :personal, :value "z1"} {:type :personal, :value ""}]
[{:type :personal, :value "ZG"} {:type :personal, :value "N"}]
[{:type :personal, :value "qQ6"} {:type :work, :value "A8h"}]
[{:type :personal, :value "2588"} {:type :personal, :value "Djv12"}]
[{:type :work, :value ""} {:type :personal, :value "c"}]
[{:type :work, :value "U2uHMI"} {:type :personal, :value "376ihr"}]
[{:type :personal, :value "n6Ct4d"} {:type :personal, :value "V09xL8"}]
[{:type :work, :value "9"} {:type :personal, :value ""}])#2020-12-2308:10ikitommiif the first map is always work and second personal, :tuple for the win:
[:tuple
[:map [:type :work] [:value string?]]
[:map [:type :personal] [:value string?]]]
#2020-12-2308:22elaroussThank you, the :multi works for my use case 🙏#2020-12-2308:08ikitommiif the :type defines the shape of the data, you can try :multi:
(require '[malli.generator :as mg])
(mg/sample
[:vector {:min 2, :max 2}
[:multi {:dispatch :type}
[:work [:map [:type [:= :work]] [:value string?]]]
[:personal [:map [:type [:= :personal]] [:value string?]]]]])
(mg/sample
[:tuple {:registry
{::element
[:multi {:dispatch :type}
[:work [:map [:type [:= :work]] [:value string?]]]
[:personal [:map [:type [:= :personal]] [:value string?]]]]}}
::element ::element])#2020-12-2308:42borkdude@ikitommi In other languages people usually define this as an ADT. Maybe :enum can also be used for this?#2020-12-2308:43borkdude(I didn't see any docs about :enum in the README, only examples)#2020-12-2308:43ikitommi:enum is about values, :or is about schemas:
[:or
[:map [:type [:= :work]] [:value string?]]
[:map [:type [:= :personal]] [:value string?]]]
fixed, thanks to @borkdude#2020-12-2308:44borkduderight, that's what I was thinking#2020-12-2308:44ikitommibut, :or does a linear scan over schema, :multi can be much more efficient.#2020-12-2308:45borkdudewhy did you have to write [:work [:map [:type [:= :work]] [:value string?]]] with :multi but [:map [:type :work] [:value string?]] with :or?#2020-12-2308:45ikitommi(`either` was deprecated in Plumatic Schema because of this, malli supports that as it’s usefull to describe the real world, despite being slow)#2020-12-2308:46ikitommicould be:
[:work [:map [:type string?] [:value string?]]]
#2020-12-2308:46borkdudeif you support :case for a closed world of possibilities it could possibly be even faster?#2020-12-2308:46ikitommi… but the generator would then just generate any keyword for the :a: #2020-12-2308:47ikitommiwhat kind of code would the :case emit?#2020-12-2308:47borkdudeI was referring to [:type [:= work]] vs [:type :work]#2020-12-2308:47ikitommioh, that’s my bug. [:type :work] doesn’t work.#2020-12-2308:47ikitommilooks for :work from the registry, which doesn’t exist.#2020-12-2308:48borkdudeah ok then#2020-12-2308:48borkdude(I have no idea what :case would emit, just bouncing an idea)#2020-12-2308:48ikitommi[:= :work] is basically same as [:enum :work]#2020-12-2308:48borkdudeso [:map [:enum :work :personal] [:value string?]] would also work#2020-12-2308:49ikitommi[:map [:type [:enum :work :personal]] [:value string?]] yes#2020-12-2308:50ikitommibut if the :type effects the :value , or any other parts of the schema, then :multi is the way.#2020-12-2308:51borkdudedoes it generate a multi-method behind the scenes?#2020-12-2308:51ikitommino, it’s closed and immutable by design.#2020-12-2308:52ikitommijust a dispatch map from key -> schema. and the key is looked up using the :dispatch function.#2020-12-2308:52ikitommi:dispatch function is applied to value, which returns the :multi key, which selects the schema#2020-12-2308:53ikitommiand, :dispatch can be a sci-function, of course 🙂#2020-12-2308:53ikitommi(m/validate
[:multi {:dispatch 'first}
[:sized [:tuple keyword? [:map [:size int?]]]]
[:human [:tuple keyword? [:map [:name string?] [:address [:map [:country keyword?]]]]]]]
[:human {:name "seppo", :address {:country :sweden}}])
; true#2020-12-2308:54ikitommiso, there is always a map-lookup, so not as fast as case, but it would require code-generation, which would mean a macro.#2020-12-2308:55borkdudeyeah, that makes sense#2020-12-2308:57ikitommicould add a :conditional, where the entry keys are functions, and the first one will match. Like :or, but more explicit and short-circuits on first match:
[:conditional
[map? [:map [:x int?]]]
[int? [:int {:min 1, :max 2}]]]#2020-12-2308:58ikitommimalli.core is distilled into tons of helper functions to build new schemas, really easy to add things like this atm.#2020-12-2308:59borkdudedoesn't :or short-circuit?#2020-12-2309:01borkdudebtw, I also implemented case as a map-lookup in sci, same problem ;)#2020-12-2309:04ikitommisorry, it does. I think the issue is that you can have multiple branches that would match, making the rest effectively unreachable.#2020-12-2309:06ikitommialso, the :or tranformation is bit hacky, depending whether you are doing decoding or encoding, we need to check if the transformation turned the value valid or not. that’s slow.#2020-12-2309:06ikitommi… but, works 🙂#2020-12-2602:23steveb8nI’ve started switching out Spec for Malli to improve performance in hot code. it works great except when using recursive schemas. I’ve created a gist to illustrate https://gist.github.com/stevebuik/e4f3475e46dd5ebb1de7707438fa073f#2020-12-2602:24steveb8nanyone seen this before? Or am I making a mistake somehow?#2020-12-2609:56ikitommi@steveb8n you are using :and, which composes the two maps - in validation it runs two validations, one for each map. If you use malli.util/merge, the end result is one schema, and validation runs it just once. tested the first example, it’s 90ns vs 97ns, which is the cost of iterating over 2 MapEntrys instead of just one.
(require '[malli.util :as mu])
(def simple-tree-node
(mu/merge
simple-node
[:map {:registry {::child simple-node}}
[:children {:optional true} [:vector [:ref ::child]]]]))#2020-12-2611:35steveb8nNow that's a Xmas gift! I'll try it tomorrow. Thanks#2020-12-2610:01ikitommione note about merge: both the properties and children are merged (no deep merge atm), so the last :registry property wins. hadn’t thought of that :thinking_face:#2020-12-2610:01ikitommibut, here, it’s:
simple-tree-node
;[:map
; {:registry #:recursive-malli-perf{:child [:map [:name string?]]}}
; [:name string?]
; [:children {:optional true} [:vector [:ref :recursive-malli-perf/child]]]]#2021-12-2702:43euccastrosorry if I missed something basic, but is there a way to specify that a string field should match a regular expression, without resorting to SCI functions?#2021-12-2707:51steveb8nhave you considered using :fn without using sci? why is that not a good solution?#2021-12-2713:20ikitommihttps://github.com/metosin/malli/blob/master/test/malli/core_test.cljc#L484-L509#2021-12-2713:20ikitommiany documentation PRs most welcome#2021-12-2715:23euccastroPR here: https://github.com/metosin/malli/pull/323/files#2021-12-2715:41ikitommithanks!#2021-12-2704:46steveb8n@ikitommi can I trouble you for one more gist comment? this is not urgent so I’m happy to wait until your status is not “vacationing” 🙂 https://gist.github.com/stevebuik/e63735d99fca94041120f9b0e25b616d#2021-12-2705:05steveb8nnot urgent because I can work around the perf by not using recursion in my initial spec replacements. full recursion will be useful but can come later#2021-12-2705:05steveb8nand because OSS should never be urgent 🙂#2021-12-2706:23steveb8nI wonder if I/we could turn this into a useful sample for others to learn from? happy to do this if you agree it would be useful#2021-12-2713:24ikitommithanks for the awesome repro, easy to dig in to this when have the extra time.#2021-12-2801:24steveb8nThank you for such a big upgrade to Spec 🙂 BTW I’m also noticing some very slow perf when compiling medium complexity nested schemas. Are you interested in test cases for that as well?#2021-12-2804:42steveb8nThe compilation slowness appears to be caused by liberal use of :merge. when I remove it and just compose maps (i.e. inline) I see compile speeds improve by 10x#2021-12-2805:03steveb8nI’ve worked around both the compilation (i.e. validator) and runtime perf issues so this is not urgent for me. I’ll be happy to provide more test cases if you need them#2021-12-2718:27borkdudeHow does malli expect bytes? to be delivered e.g. when you have a JSON api#2021-12-2718:27borkdudeas base64? (that would seem random)#2021-12-2718:28borkdudeor would one do a pass over the json, deserialize the base64 manually?#2021-12-2718:28borkdudeI'm asking this since babashka pods communicate via edn or json, which doesn't have a built-in way to represent bytes. transit does have it#2021-12-2719:27ikitommi@borkdude atm, there is no default encoding/decoding bytes and strings. Looking at OpenAPI docs, the guide is:
> base64-encoded characters, for example, U3dhZ2dlciByb2Nrcw==#2021-12-2719:28ikitommihttps://swagger.io/docs/specification/data-models/data-types/#2021-12-2719:28borkdudefor now I went with transit. it does it for me.. I should look into how it does it#2021-12-2719:30borkdude#2021-12-2719:29ikitommiso, there could be a byte<->string transformers in malli.transform to make this a default#2021-12-2723:12steveb8nIs this a bug? If so, I will log it….
(m/validator [:schema
{:registry (merge
(m/base-schemas)
{:person [:multi {:dispatch :person/gender}]})}
:person])
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1}#2021-12-2723:16steveb8nworkaround….#2021-12-2723:17steveb8n(m/validator [:schema
{:registry {:person [:multi {:dispatch :person/gender}]}}
:person])#2021-12-2916:16aaron51How do I describe keywords with a certain namespace? Something like [:qualified-keyword {:namespace :asdf}] to match :asdf/x and :asdf/y#2021-12-2916:55ikitommi@aaron51 something like [:and :qualified-keyword [:fn (fn [x] (= :asdf (namespace x))]]?#2021-12-2916:56aaron51yes! thank you#2021-12-2916:58ikitommiYou can also create your own :qualified-keyword impl quite easily, on top of malli.core/-simple-schema, which would read the :namespace property and validate it behind the scenes. e.g. the syntax you proposed. Not near computer, but there are examples how to do that kind of stuff in the user space.#2021-12-2916:59ikitommiAlso, sounds useful enough, so you could do a PR of that.#2021-12-2917:00aaron51thank you, I’ll give it a try 👍#2021-12-2917:04ikitommiSome examples how to build custom s
(simple) schemas: https://github.com/metosin/malli/blob/master/test/malli/core_test.cljc#L1392-L1470#2021-12-3018:56Michael WIs it possible to take a swagger file and generate a malli schema? I see how to convert schema to swagger, but not the reverse in the docs.#2021-12-3020:23ikitommi@michael819 currently, no. There is a stalled PR of JSON Schema -> malli conversion. I would guess it takes days / weeks to make it complete enough, don't have the time myself for that. But, contributions most welcome.#2021-12-3020:44Michael WI think it's a chicken and egg problem. Should I create a malli schema for openapi to validate the files, then work from there? Just trying to gauge the amount of work necessary and a general idea of where to start.#2021-12-3120:17eoliphantHi, I’m using malli ‘spec’ishly’ with the mutable custom schema suggested in the readme. One annoyance I’m running into is the need to set the malli.registry/type to custom as a JVM property in quite a few places. I’n the current case, I’m running a Fulcro app as a Datomic ion. I ended up needing to set the property to run shadow-cljs itself (not just setting it as a closure define), the ion cmd line tools, etc, and I’m calling System/setProperty directly in code in the server entry points to try to make sure it’s set there as you can’t really set system props for the server runtime. Just wondering if longer term there might be a better way to handle this#2021-01-0116:57ikitommi@eoliphant the first alpha wanted to be immutable by default. We could revisit that for 1.0.0, starting to think it would be ok if malli was optionally immutable: the JVM/compiler option could be to close things up.#2021-01-0117:01ikitommiwhen one does multi-tenant schema systems, anyway the registry is either local or passed as option -> one can have already full control of the registries in places where it matters. The system default registry is just for your own core app, not the tenants.#2021-01-0117:01ikitommimaybe have a up-to-date branch of malli with the option reversed?#2021-01-0117:04ikitommialso, comments welcome on having an optional global registry. Something like (malli.mutable/register! :user/name :string)#2021-01-0202:35y.khmelevskiiHi! I would like to ask about feature request for malli.generator.
It would be great to support relations and constrains like in https://github.com/reifyhealth/specmonstah
For example
(def schema
{:user {:prefix :u
:spec ::user}
:post {:prefix :p
:spec ::post
:relations {:created-by-id [:user :id]}}
:like {:prefix :l
:spec ::like
:relations {:post-id [:post :id]
:created-by-id [:user :id]}
:constraints {:created-by-id #{:uniq}}}})
Is it make sense for malli roadmap?#2021-01-0209:41ikitommi@U1GTUPAVB specmonstah does a lot of things and it might be generic enough to support malli easily. Would you like to investigate / suggest that to specmonstah maintainers?#2021-01-0209:44ikitommibut, please write the issue to malli repo, so it's on potential backlog. The idea is good, would use that myself.#2021-01-0209:45ikitommiRelated: https://github.com/metosin/malli/issues/53#2021-01-0622:37steveb8nI use specmonstah so I'm interested in this issue as well#2021-01-0700:12pithylessFYI - There is a fork of specmonstah meant to support malli, but I haven’t been tracking it so not sure how baked it is: https://github.com/lambdaisland/specmonstah-malli#2021-01-0700:19steveb8nmaybe we should suggest a name change for the fork 🙂#2021-01-0611:20ordnungswidrigGive a definition like
(def Foo [:schema {:registry {"Foo" [:map [:bar pos-int?} "Foo"])
Is there a way to “resolve” that schema to the actual schema referenced by “Foo”?
I want to programatically access the actual definition of “Foo”, e.g. to enumerate the map fields. I’ve look through the properties, options, children and reference of the schema created by (m/schema Foo) but I’m a little lost.#2021-01-0617:01ikitommi@ordnungswidrig try (m/deref Foo)#2021-01-0617:02ikitommithere is also m/deref-all, which peels all top-level RefSchemas.#2021-01-0617:04ikitommialso, you can always m/walk the schema with option :malli.core/walk-refs, which walks over all refs. Not sure of that one blows up with recursive schemas...#2021-01-0617:39ordnungswidrigOh, i had tried ref but not deref.#2021-01-1115:11hansbuggeNone of the comparator schemas have default error messages, e.g.,
(-> (m/explain [:> 0] -1)
:errors first me/error-message)
;; => "unknown error"
Is there a good reason why some simple error functions haven't been added to malli.error/default-errors other than "no-one has done it yet"? 🙂#2021-01-1115:40hansbuggeI've opened a PR with suggestions for some simple error functions https://github.com/metosin/malli/pull/332#2021-01-1118:40ikitommiMerged the PR, thanks!#2021-01-1518:47ikitommi🚀 🚀 🚀 regex schemas merged in master! feel free to test: validate, explain, transform and generate - parsing/conform still wip.#2021-01-1518:55ikitommithanks to @nilern, the impl is quite snappy, currently order(s) of magnitude faster than spec & seqexp.#2021-01-1700:04lreadI’m trying out malli for the first time. I’m using it to validate user entered options. I really like the humanize spell check support. Check out this helpful error message!
Invalid inline options: #:test-doc-blocks{:read-cond ["should be spelled :test-doc-blocks/reader-cond"]}
file: doc/design/namespaced-elements.adoc line: 277#2021-01-1700:04lreadAnyhoo, thanks!#2021-01-1718:50naomarikI've yet to update malli since before it had an actual version number. Looks way more feature rich now.#2021-01-1719:13ikitommithe function schemas are in master, but not released. there is a bug with generating multi-arity functions and as the sequence/regex schemas are in, will add support for varags. Then good to release a new version. Hopefully much more and smaller releases after that.#2021-01-1719:14ikitommibtw, comments welcome on the malli.error/humanize output: few options how to fix that: https://github.com/metosin/malli/pull/333#discussion_r559224770#2021-01-1819:48Samuel McHughHello, I'm trying out Malli and I'm wondering what is the idiomatic approach to spec'ing a finit explicit set of values. Here's an example of a schema which works.
(m/validate
[:or [:= :foo]
[:= :bar]
[:= :baz]]
:foo)
;=> true
The syntax isn't as compact as I'd expect though. I'd hope something like
(m/validate
#{:foo :bar :baz}
:foo)
could work but I haven't seen this in the docs anywhere. Do I have to set up a custom registry for this?#2021-01-1820:02ikitommi@smchugh230395 try [:enum :foo :bar :baz]. Doc PRs welcome!#2021-01-1909:06Samuel McHughI will have a look after work today and make a doc PR. Thanks!#2021-01-1917:56Samuel McHughI'm going to spend more time understanding what's available and possible before I submit any doc changes. There is a lot to chew on!
Is there a native capacity to enforce uniqueness of items in a collection?#2021-01-2009:18ordnungswidrig@smchugh230395 can you use a set as the collection type?#2021-01-2009:20Samuel McHughThat's a good idea but there is an order on the elements of the collection in my particular use-case. I can set up other ways of ensuring that only unique items enter the collection though so it isn't critical that Malli supports it out of the box.#2021-01-2009:46ordnungswidrigI see. So “ordered set” is what you need I guess.#2021-01-2009:52ordnungswidrig(m/validate [:and [:sequential any?] (fn [c] (= (count c) (count (set c)))) something like this?#2021-01-2014:22Samuel McHughgood one, I think i'll go with this#2021-01-2014:51ordnungswidrigPretty sure this one of the least optimized way to check for distinct elements (reduce (fn [xs x] (if (xs s) (reduced false) (conj xs s))) #{} coll) might be faster.#2021-01-2010:46ikitommiJSON Schema has “uniqueItems” attribute, I guess sequences could have :distict boolean property?#2021-01-2010:46ikitommihttps://json-schema.org/understanding-json-schema/reference/array.html#uniqueness#2021-01-2010:47ikitommi… would be [:sequential {:distinct true} any?]#2021-01-2011:32ordnungswidriginteresting#2021-01-2014:09emccueI am somewhat meh on the coupling to json schema#2021-01-2014:10emccueis there some usecase or justification i am missing?#2021-01-2119:06Daniel MiladinovHi everyone, long-time listener, first-time caller here.
I’m still kind of new to malli and I’m trying to develop a spec to validate a vector of strings, but some of the strings have different validations than the others. Looking at the https://github.com/metosin/malli#sequence-schemas, it seemed to me that I should use a sequence schema with something like [:cat …].#2021-01-2119:07Daniel MiladinovBut I’m having a problem. According to the README, I should be able to do this:
(m/validate [:cat string? int?] ["foo" 0]) ; => true
But in my repl, I get this:
(require '[malli.core :as m])
=> nil
(m/validate [:cat string? int?] ["foo" 0])
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :cat}#2021-01-2119:23Daniel MiladinovThe project I’m working in has reitit 0.5.11 as a direct dependency, which appears to pull in malli 0.2.1 as a transitive dependency.#2021-01-2120:02ikitommiif you use deps, you can take a dependency to the latest commit of Malli, which has the sequence schemas. Will try to ship a Big 0.3.0 out within few weeks with those officially in.#2021-01-2120:03ikitommiThere is also [:tuple string? int?] for fixed length vectors with schema for each element#2021-01-2120:03Daniel MiladinovThank you!!!#2021-01-2120:03Daniel MiladinovThat will serve my purposes in the meantime#2021-01-2315:14ikitommim/conform landed in master by @nilern! #343 will change the names to m/parse and m/parser:
(m/parse
[:* [:cat*
[:prop string?]
[:val [:alt*
[:s string?]
[:b boolean?]]]]]
["-server" "foo" "-verbose" true "-user" "joe"])
;[{:prop "-server", :val [:s "foo"]}
; {:prop "-verbose", :val [:b true]}
; {:prop "-user", :val [:s "joe"]}]#2021-01-2410:15ikitommiby the power of generic schema walking (answer to #343):
(require '[malli.core :as m])
(defn normalize-properties [?schema]
(m/walk
?schema
(fn [schema _ children _]
(if (vector? (m/form schema))
(into [(m/type schema) (m/properties schema)] children)
(m/form schema)))))
(normalize-properties
[:map
[:x int?]
[:y [:tuple int? int?]]
[:z [:set [:map [:x [:enum 1 2 3]]]]]])
;[:map nil
; [:x nil int?]
; [:y nil [:tuple nil int? int?]]
; [:z nil [:set nil
; [:map nil
; [:x nil [:enum nil 1 2 3]]]]]]#2021-01-2510:43ikitommim/parse and m/parser now in master. Feedback and test reports most welcome.#2021-01-2514:45kwrooijenm/parse looks great! I think that's going to clean up a lot of my code once I get a chance to use it#2021-01-2518:34ikitommiupdated README with parsing examples: https://github.com/metosin/malli#parsing-values#2021-01-2518:36ikitommithe hiccup example is from minimallist, thanks to @U8MJBRSR5 for the original example 🙂#2021-01-2518:39Vincent CantinI like it 🙂#2021-01-2518:56Vincent Cantin@U055NJ5CC in Malli’s version, there is no indication that the :node is a vector. Was it intentional?#2021-01-2519:23ikitommiOh, that is missing. It accepts all sequences#2021-01-2519:24ikitommican't compose with :and here, need to constraint in some other way#2021-01-2519:35Vincent Cantinwithout the vector indication, the generator might create a list, that's a problem.#2021-01-2519:37ikitommiyou can add :gen/fmap vec to gen a vec, but having a :kind etc. property would make it work in all the places (validate, gen, transform)#2021-01-2518:35ikitommi:or, :cat and :alt all use non-named branches, :or* , :cat* and :alt* are the named variants.#2021-01-2518:36ikitommiif somenone has better names for those, please suggest.#2021-01-2518:42Vincent CantinNaming things is hard, indeed.#2021-01-2519:09oliverHi! I'm using malli for the first time on a project where I have to derive HTML-Forms from the body-structure of http-requests. So far the experience has been great – and much more natural than with spec.
Now I feel the need for something I was unable to find in the Readme. Say I have the following schema definition:
[:map {:closed true} [:a int? :b int?]]
I would like to further constrain :b depending on :a, e.g.:
- :b must be greater than :a
— or —
- :b should only be present iff :a is greater than 3 (nil otherwise)
Is there a way to express this without recourse to [:fn ] schemas?
I'll be thankful for any hints on this!#2021-01-2520:56emccuehow would I annotate a function for runtime checking at dev time?#2021-01-2521:19ikitommi@emccue no such thing yet, but should be easy to add. contributions welcome. https://github.com/metosin/malli/issues/349#2021-01-2521:20ikitommi@services :and & :fn is the way to do it now.#2021-01-2521:21ikitommihttps://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D#2021-01-2717:59oliverHi, sorry for taking so long to respond… many thanks for the playground example… I wasn't even aware that existed. I'll be using [:fn …] then. straightforward!#2021-01-2812:50danielnealIs there a way of converting malli definitions to swagger 3 format?#2021-01-2815:46juhoteperiNot sure about Swagger 3, malli.swagger might be do Swagger 2, but not sure if there is difference in this level. It is just JSON Schema with a few extensions.#2021-01-2816:51danielnealI've been reading the swagger specification and it's massive#2021-01-2816:51danielnealhttps://swagger.io/specification/#2021-01-2816:53danielnealThe most obvious difference is that in JSON Schema and Swagger 2, there is a "definitions" key for common objects, whereas this doesn't exist in OpenAPI 3#2021-01-2816:54danielnealI think it's replaced by the components object, or the schemas property on the components object#2021-01-2816:56juhoteperiWith currently implementation, Malli only handles the schema -> properties part. Reitit/Ring-swagger/Compojure-api does the higher level Swagger2 generation.#2021-01-2816:57juhoteperiThe properties part (JSON Schema) probably doesn't change much if at all between Swagger2 / OpenAPI 3.#2021-01-2817:07danielnealdo you know if there are plans to support OpenAPI 3 in reitit?#2021-01-2817:08danielnealah yeah, there's an open issue https://github.com/metosin/reitit/issues/84#2021-01-2813:00ikitommiJust chatted with @juhoteperi on Metosin internal about that. should be, not atm. should be a thin layer on top of malli.json-schema.#2021-01-2813:01ikitommiwho would like to contribute those?#2021-01-2813:12danielnealhaha great timing#2021-01-2813:39danielnealah oh you mean you chatted with him after my comment 🙂#2021-01-2813:39danielnealI could have a look at it I guess, but swagger and malli are both completely new to me#2021-01-2813:40ikitommiJust = yesterday. https://github.com/metosin/malli/pull/354#2021-01-2815:10ikitommithis is going to be so good.#2021-01-2815:11ikitommi#2021-01-2815:46juhoteperiNot sure about Swagger 3, malli.swagger might be do Swagger 2, but not sure if there is difference in this level. It is just JSON Schema with a few extensions.#2021-01-2910:43danielnealwhat order do :and schemas operate in#2021-01-2910:43danielnealis it possible to do
(m/explain [:and
int?
(m/-simple-schema
{:pred #(> % 3)})] "asdf")#2021-01-2910:44danielnealand have the int? check prevent the pred from running and erroring?#2021-01-2911:02danielneallike in spec
(s/def ::greater-than-3
(s/and int? #(> % 3)))#2021-01-2911:13dharriganThis will work...#2021-01-2911:13dharrigan(def foo [:and int? [:fn (fn [x] (> x 3))]])#2021-01-2911:13dharrigan#'user/foo
user=> (m/validate foo "abcd")
false
user=> (m/validate foo 1)
false
user=> (m/validate foo 3)
false
user=> (m/validate foo 4)
true
#2021-01-2911:16dharriganYou can, of course, shorten it, i.e., (def foo [:and int? [:fn #(> % 3)]])#2021-01-2911:18danielnealinteresting, I wonder why :fn works but -simple-schema doesn't#2021-01-2911:19dharriganIt's in the documentation to use that 🙂#2021-01-2911:19dharriganYou can try it out here as well #2021-01-2911:26danielnealcool, I'll use :fn 🙂#2021-01-2911:29dharriganActually, it works with simple-schema too#2021-01-2911:30dharriganuser=> (def foo2 [:and int? (m/-simple-schema {:pred #(> % 3)})])
#'user/foo2
user=> (m/validate foo2 4)
true
user=> (m/validate foo2 3)
false
user=> (m/validate foo2 1)
false
user=> (m/validate foo2 "and")
false
user=> #2021-01-2911:30dharriganI missed that, so I suppose you can use that as well 🙂#2021-01-2911:31dharriganchoices...choices...#2021-01-2911:41danielneal@dharrigan, validate succeeds, but explain fails#2021-01-2911:41danielnealcould be a bug perhaps?#2021-01-2911:41danielneal(m/explain foo2 "and") => java.lang.ClassCastException#2021-01-2912:08danielnealI guess it does makes sense that explain on an :and would give all the reasons something fails, but it doesn't make sense -simple-schema fails where :fn doesn't.#2021-01-2912:09juhoteperiIf you are just checking > you could also use [:int {:min 3}] or [:and int? [:>= 3]].#2021-01-2912:09juhoteperiFirst one works best with JSON Schema#2021-01-2912:12danielnealah, the actual example here is a bit different, it's for a kebab case keyword#2021-01-2912:13danielneal[:and
{:description "A keyword joined by hyphens"
:swagger/description "A string joined by hyphens"
:swagger/example "kebab-case-string"
:error/message "should be a kebab-case keyword"}
keyword?
[:fn kebab-case-keyword?]]#2021-01-2912:14danielneal(where in this case kebab-case-keyword? fails on non-keywords)#2021-01-2915:08ikitommiWIP: Schemas of Schemas:
(-> (m/-simple-schema
{:type :int
:pred string?
:properties-schema [:map [:min nat-int?] [:max nat-int?]]
:property-pred (m/-min-max-pred nil)})
(m/properties-schema))
; => [:map [:min nat-int?] [:max nat-int?]]
also:
(let [OneOf (m/-simple-schema
(fn [{:keys [count]} values]
(let [value? (set values)]
{:type :user/over
:pred value?
:properties-schema [:map [:count [:= count]]]
:children-schema (into [:tuple] (map (fn [x] [:= x]) values))
:type-properties {:error/message (str "should be one of " values)
:json-schema {:type "oneOf", :values values}}})))
schema (m/schema [OneOf {:count 6} 1 2 3 4 5 6])]
{:properties-schema (m/properties-schema schema)
:children-schema (m/children-schema schema)})
;{:properties-schema [:map [:count [:= 6]]]
; :children-schema [:tuple [:= 1] [:= 2] [:= 3] [:= 4] [:= 5] [:= 6]]}#2021-02-0313:05danielneallol, trolling hard rich#2021-02-0315:52pbailleHi, i'm playing with malli and am confused by this example from the README:
(m/validate [:? int?] [1 2])
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :?}#2021-02-0316:23ikitommi@pbaille README describes what is in the master, sequence/regex schemas are not in official release. If you take the latest commit as dependency via deps, there are there#2021-02-0316:24ikitommiwill cut a release soon, work needed for the function schemas still#2021-02-0316:31pbaille@ikitommi thank you a lot 🙂#2021-02-0414:27dakraHow can I read the properties of attributes in a map. E.g. the description of one-attr in this example
(def test-schema
(malli/schema [:map {:description "test schema desc."}
[:one-attr {:description "attr description"} string?]]))
(malli/properties test-schema) ;; => {:description "test schema desc."}
(malli/properties (mu/get test-schema :one-attr)) ;; => nil
#2021-02-0416:25ikitommi@daniel415 try (mu/find schema key) and you can iterate the child entries with m/children#2021-02-0416:30dakraThe mu/get does work and gives me the correct schema back but just m/properties is not returning anything. (only for maps)
And m/children is not working in my example?
(-> (malli/schema [:map {:description "test schema desc."}
[:one-attr {:description "attr description"} string?]])
(mu/find :one-attr)
malli/children)
return this error
Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:12).
:malli.core/invalid-schema {:schema :one-attr}#2021-02-0416:32ikitommitry removing the last m/children#2021-02-0416:33ikitommimu/find should return the entry tuple, props being the second value#2021-02-0416:35ikitommialso, this should hold:
(-> [:map [:x [:int {:default 1}]]]
(m/get :x)
(m/properties))
;; -> {:default 1}#2021-02-0416:35ikitommicoding blind from a phone, might contain errors :)#2021-02-0416:36ikitommibut: maps, map entries and map entry values can all have properties#2021-02-0416:37dakraHmm, your example works. But when I simply have string? or so it doesn't work.
(-> [:map [:x {:default 1} string?]]
(mu/get :x)
(m/properties))
;; => nil#2021-02-0416:38dakraSo this is the equivalent that works:
(-> [:map [:x [:string {:default 1}]]]
(mu/get :x)
(m/properties))
;; => {:default 1}#2021-02-0418:52ikitommistring? doesn’t have properties. is is short format for [string?]. You can add properties to it too, e.g. [string? {:default 1}]. Full example:
(-> [:map [:x [string? {:default 1}]]]
(mu/get :x)
(m/properties))
;; => {:default 1}
(still not at malli repl, just guessing what it returns)#2021-02-0418:58ikitomminow at the repl:
(def Schema
[:map {:in "map"}
[:x {:in "entry"} [:string {:in "value"}]]])
(m/properties Schema)
;; => {:in "map"}
(-> Schema
(mu/find :x)
(second))
;; => {:in "entry"}
(-> Schema
(mu/get :x)
(m/properties))
;; => {:in "value"}'#2021-02-0419:00ikitommisome helpers:
(defn entry-properties [schema key]
(-> schema (mu/find key) (second)))
(defn value-properties [schema key]
(-> schema (mu/get key) (m/properties)))
(m/properties Schema) ;; => {:in "map"}
(entry-properties Schema :x) ;; => {:in "entry"}
(value-properties Schema :x) ;; => {:in "value}#2021-02-0419:44dakraOK, thank you very much. Now it's clear. I wasn't aware that string? is shorthand for [string?] and also didn't really think about the difference between entry and value properties. Makes sense now 🙂#2021-02-0609:45ikitommim/unparse and m/unparser landed in master! parse + unparse with hiccup:
(def Hiccup
[:schema
{:registry {"hiccup" [:or*
[:node [:cat*
[:name keyword?]
[:props [:? [:map-of keyword? any?]]]
[:children [:* [:schema [:ref "hiccup"]]]]]]
[:primitive [:or*
[:nil nil?]
[:boolean boolean?]
[:number number?]
[:text string?]]]]}}
"hiccup"])
(m/parse
Hiccup
[:div {:class [:foo :bar]}
[:p "Hello, world of data"]])
;[:node
; {:name :div,
; :props {:class [:foo :bar]},
; :children [[:node
; {:name :p
; :props nil
; :children [[:primitive [:text "Hello, world of data"]]]}]]}]
(->> [:div {:class [:foo :bar]}
[:p "Hello, world of data"]]
(m/parse Hiccup)
(m/unparse Hiccup))
;[:div {:class [:foo :bar]}
; [:p "Hello, world of data"]]#2021-02-0609:49ikitommitwo small things for 0.3.0 with all the stuff in.#2021-02-0610:20borkdudenice!#2021-02-0610:20borkdudethis is like conform right?#2021-02-0610:40ikitommiyes, same as s/confom and s/unform. At the moment, there are no property-based custom parsers/unparsers that a user could add (like custom property based transformers) and in case of error, just ::m/invalid is returned. I think we can later combine -parse, -unparse and -explain implementations together, so that one can see partially parsed results and get more details when parsing/unparsing fails (same output as in -explain).#2021-02-0708:41ikitommiNot sure how useful the function schema checking is in real life (maybe for tooling?), but here’s the wip:
(require '[malli.generator :as mg])
(def check
(mg/function-checker
[:function
[1 [:=> [:cat :int] :int]]
[2 [:=> [:cat :int :int] [:int {:max 10}]]]]))
(check
(fn
([x] x)
([x y] (mod (+ x y) 10))))
; => nil
(check
(fn
([x] x)
([x y] (+ x y))))
;({:total-nodes-visited 18,
; :depth 6,
; :pass? false,
; :result false,
; :result-data nil,
; :time-shrinking-ms 0,
; :smallest [(0 11)],
; :malli.core/schema [:=> [:cat :int :int] [:int {:max 10}]]})
(check
(fn
([x y] (mod (+ x y) 10))))
;({:total-nodes-visited 0,
; :depth 0,
; :pass? false,
; :result "Wrong number of args (1) passed to: user/eval97462/fn--97463",
; :time-shrinking-ms 0,
; :smallest [(0)],
; :malli.core/schema [:=> [:cat :int] :int]})#2021-02-0807:16ikitommithis @UG9U7TPDZ, in master, will be released soon.#2021-02-0709:01caumondhi, I search a little bit long before finding that representing date in malli could be easily done with inst? Is it a recommended approach? I have a doubt as I would expect to find a reference of that in the malli doc which I did not.#2021-02-0711:20ikitommiinst? is ok, but see https://github.com/metosin/malli/issues/49#2021-02-0717:12caumondYes, I saw the PR, it was not completely enlightening. I understood something is missing, some of the proposal, but I did not understand there what is the recommend approach awaiting for that PR to be merged.#2021-02-0717:13caumondI keep the "`inst` is ok" part !#2021-02-0711:22ikitommi:not merged in master:
(mg/sample [:not :string])
;([]
; #{}
; nil
; \$
; nil
; nil
; {#uuid"3565d7f3-5561-439f-9368-5bfdbd03fc8e" -2/3}
; ()
; [\^]
; [[-110317951337N \:] #{2.15625 R+3/qfp}])#2021-02-0807:10kwrooijenHow do you spec a function? e.g. I'd like to do this:
[:map
[:component/foo 'fn?]]
#2021-02-0807:10kwrooijenBut that results in :malli.core/invalid-schema {:schema fn?}#2021-02-0807:14ikitommi@kevin.van.rooijen you can say: [:map [:component/foo [:fn fn?]] or 'fn? to keep it serializable#2021-02-0807:15ikitommimy code sniplet from yesterday show how one can enforce input & output params and arities too.#2021-02-0807:20kwrooijenCool, [:fn 'fn] works like a charm#2021-02-0807:20kwrooijenThanks, looking forward to the next release 🙂 Will be replacing all my specs with Malli once that happens#2021-02-0813:41ikitomminice. would be great if some non-metosinian blogged about the goods and bads of spec, schema and malli - from real life experience. happy to try fix all the bads in malli 😉#2021-02-0820:01kwrooijenI might do that eventually, although I have a hard time thinking of bad things about Malli, so it'll probably only be success stories 🙂#2021-02-0820:03kwrooijenI'm currently using it for Gungnir, and working on integrating it for Duct / some web development stuff. At my job we heavily use Malli as well (As well as serialized through an API)#2021-02-0820:04kwrooijenI might use the new parser for a (hiccup) templating language that I'm building. So that's pretty cool timing of the new feature 😄#2021-02-0911:48danielnealWhat's the best way to define a string format so that it comes through to swagger#2021-02-0911:49danielneal(swagger/transform [:map
[:a {:swagger/format "date-time"} string?]])
; => {:type "object", :properties {:a {:type "string", :format "date-time"}}, :required [:a]}#2021-02-0911:49danielnealthis works, but I'm wondering if there's a better way that also has validation#2021-02-0911:51ikitommimaybe split it:
(def DateTimeString [:string {:swagger/format "date-time"}])
(swagger/transform [:map [:a DateTimeString]])
; => {:type "object", :properties {:a {:type "string", :format "date-time"}}, :required [:a]}#2021-02-0911:55ikitommiwhat do you mean by “also has validation”?#2021-02-0911:56danielneallike maybe a [:fn #(java.time.Instant/parse %)] or something?#2021-02-0911:57ikitommithat would return an Instant, not string.#2021-02-0911:59ikitommiafter the date schemas are implemented, it would be:
(swagger/transform [:map [:a :instant]])
; => {:type "object", :properties {:a {:type "string", :format "date-time"}}, :required [:a]}
#2021-02-0911:59danielnealah, that would be ideal#2021-02-0912:00ikitommibut, do you want Instant as a result or just a string that is formatted like an instant?#2021-02-0912:04danielnealhaha that's a good question. For consumers of the api, and in the docs etc it should be a string that is formatted like an instant. It would be nice if when we coerce parameters, we could get back a real instant.#2021-02-0912:04danielnealIs what :string/decode is used for#2021-02-0913:53ikitommi@danieleneal if you want a full custom type, here’s a sample:
(ns demo
(:require [malli.core :as m]
[malli.error :as me]
[malli.generator :as mg]
[malli.transform :as mt]
[malli.json-schema :as json-schema]
[clojure.test.check.generators :as gen])
(:import [java.time Instant]
[java.util Date]))
(def instant
(let [string->instant #(if (string? %) (Instant/parse %))]
(m/-simple-schema
{:type 'instant
:pred (partial instance? java.time.Instant)
:type-properties {:error/message "should be instant"
:decode/string string->instant
:decode/json string->instant
:json-schema {:type "string", :format "date-time"}
:gen/gen (gen/fmap #(.toInstant ^Date %) (mg/generator inst?))}})))
(m/form [:map [:x instant]])
; => [:map [:x instant]]
(m/validate instant (Instant/now))
;=> true
(-> [:map [:x instant]]
(m/explain {:x "kikka"})
(me/humanize))
; => {:x ["should be instant"]}
(json-schema/transform [:map [:x instant]])
;{:type "object"
; :properties {:x {:type "string"
; :format "date-time"}}
; :required [:x]}
(m/decode
[:map [:x instant]]
{:x "2021-02-09T13:49:44.419Z"}
(mt/json-transformer))
; => {:x #object[java.time.Instant 0x5aed36d3 "2021-02-09T13:49:44.419Z"]}
(mg/generate [:map [:x instant]])
; => {:x #object[java.time.Instant 0x4878fa62 "1970-01-01T00:00:00.295Z"]}#2021-02-0913:54ikitommie.g. m/-simple-typeallows to (easily?) build custom schemas that cover all the aspect: transforming, humanized errors, generators, json-schema, etc.#2021-02-0913:55ikitommimuch of the core itself is built on top of that.#2021-02-0913:56ikitommi:type-properties allow one to hide implementation details. one could do the same fully using normal schema properties, but all the details would be in your face when you look at the schema form.#2021-02-0914:18danielnealthanks @ikitommi, that looks great#2021-02-1215:16kwrooijenHi, is this a bug?
(def model
[:map
[:my/field {:default nil} :int]])
(m/decode model params mt/default-value-transformer)
;; => {}
I'm expecting {:my/field nil} here#2021-02-1215:21ikitommi@kevin.van.rooijen a bug, PR welcome#2021-02-1215:21kwrooijenWill do 👍#2021-02-1215:21kwrooijenKind of have dejavu with this bug#2021-02-1215:21kwrooijenhttps://github.com/metosin/malli/pull/223/files lol 😅#2021-02-1215:24ikitommifalsey is hard.#2021-02-1215:24kwrooijenGood thing Clojure only has 2 falsey values.. instead of some other languages I won't name#2021-02-1422:29caumondHi guys, I have tried to draw malli's dot representation. Everything's fine until I try to print a [:<= 20]or [:< 20] . In both case, it seems to be an invalid sequence. At least tangle doesn't know how to handle it. Everything's fine if I replace [:<= 20 in the string built by transform.#2021-02-1422:29caumondIs it a PR?#2021-02-1503:23aaron51Is there an easy way to use Malli in clojure.test/is?
I’d like to assert that m/validate returns true, and show m/explain output when it fails.#2021-02-1504:27aaron51(is (validate schema data) (humanize (explain schema data))) — anything like this built in? Or just validate! that throws on failure?#2021-02-1507:55borkdudeWrite a function?#2021-02-1508:40ikitommithere are no test helpers atm, but should be easy to do in user space as borkdude suggests, Are there good helpers / utilities for this (testing) in spec? in schema?#2021-02-1522:38aaron51Yes, it is straightforward enough to write it in user space. But I thought it would be a common case… Found a few similar things:
• schema: https://github.com/plumatic/schema/blob/master/test/clj/schema/test_macros.clj
• schema: https://github.com/plumatic/schema/blob/master/test/cljx/schema/core_test.cljx
• clojure-expectations has “spec expectations” https://github.com/clojure-expectations/clojure-test#2021-02-1522:40borkdudeI also have this one: https://github.com/borkdude/respeced
This was written to test fdef specs.#2021-02-1620:04ikitommiwrote an issue out of this: https://github.com/metosin/malli/issues/369.#2021-02-1508:34ikitommi@caumond PR welcome#2021-02-1508:42ikitommiwip: function generators generate correctly function with different arities
(def f
(mg/generate
[:function
[:=> :cat int?]
[:=> [:cat int?] nat-int?]]))
(f)
;=> 132816
;=> -7823115
;=> -36
;=> -97
;=> 13412759
;=> 1444
(f 1)
;=> 1038018
;=> 11009747
;=> 8
;=> 59186626
;=> 10
;=> 5373734
(f "1")
; =throws=> :malli.generator/invalid-input {:schema [:cat int?], :args ["1"]}
(f 1 2)
; =throws=> :malli.generator/invalid-arity {:arity 2, :arities #{0 1}, :args (1 2), :schema [:function [:=> :cat int?] [:=> [:cat int?] nat-int?]]}#2021-02-1508:44borkdudeis this in a branch somewhere?#2021-02-1508:45ikitomminot yet, needs a new -min-count protocol method to regex-schma to resolve the arities from :=> schemas. few hours away from a branch.#2021-02-1508:49borkdudecool. how are you generating the fn per arity?#2021-02-1508:53ikitommipushed the current: https://github.com/metosin/malli/blob/fn_new/src/malli/generator.cljc#L104-L129#2021-02-1508:54ikitommibut that code doesn’t infer the arity, needs to set manually:
(def f
(mg/generate
[:function
[0 [:=> :cat int?]]
[1 [:=> [:cat int?] nat-int?]]]))#2021-02-1508:59borkdude:thumbsup:#2021-02-1819:56Daniel MiladinovIs it possible to use a `[:map …]` spec to describe interactions / interdependencies with map keys / values?
Like, “either this key or that key must be set”, or “these two keys must either both have empty values or both be set”#2021-02-1819:58borkdudeThis is probably done using a predicate on the map?#2021-02-1819:58borkdude(Guess)#2021-02-1819:59Daniel MiladinovI mean, I know I can write a [:fn …] spec and do whatever I need with the map validation, but I was wondering if there was anything more declarative.#2021-02-1820:49ikitommi@daniel.miladinov if you can cook up a good declarative syntax for defining the rules, I can try to summon a custom schema to do that. Declarative rule systems are not always simple.#2021-02-1820:50ikitommiNot sure if something like meander could be used to declare rules like that. Bridging Malli and Meander might be fun#2021-02-1914:24dangercoderHi! According to the docs there is a parse function which im very much interested in.
(m/parse
[:* [:cat*
[:prop string?]
[:val [:alt*
[:s string?]
[:b boolean?]]]]]
["-server" "foo" "-verbose" true "-user" "joe"])
In what namespace does this function live? I tried looking through the code and could not find it. Also tried the example (I thought, well, maybe m is bound to malli.core) 🙂#2021-02-1914:26ikitommiit’s in master, and it’s malli.core/parse.#2021-02-1914:26ikitommirelease soon, just need more tests and some docs.#2021-02-1914:27dangercoderI see its me looking in the wrong branch, sorry. I looked in 0.2.1#2021-02-2115:26ikitommiadded more info on function schema explain:
(def MyFunctionSchema
(m/schema
[:function
[:=> [:cat :int] :int]
[:=> [:cat :int :int [:* :int]] :int]]
{::m/function-checker mg/function-checker}))
(m/explain
MyFunctionSchema
(fn
([x] x)
([x y & zs] (apply max x y zs))))
; => nil
(m/explain
MyFunctionSchema
(fn
([x] x)
([x y & zs] (str (apply max x y zs)))))
;{:schema [:function
; [:=> [:cat :int] :int]
; [:=> [:cat :int :int [:* :int]] :int]],
; :value #object[],
; :errors (#Error{:path [],
; :in [],
; :schema [:function
; [:=> [:cat :int] :int]
; [:=> [:cat :int :int [:* :int]] :int]],
; :value #object[],
; :check ({:total-nodes-visited 2,
; :depth 1,
; :pass? false,
; :result false,
; :result-data nil,
; :time-shrinking-ms 0,
; :smallest [(0 0)],
; :malli.generator/explain-output {:schema :int,
; :value "0",
; :errors (#Error{:path []
; :in []
; :schema :int
; :value "0"})}})})}#2021-02-2115:29ikitommireally happy how good the function schemas are now, but still - it’s just elegant (and slow) way to verify things at runtime, e.g. not a type system.#2021-02-2115:32ikitommiin master#2021-02-2115:33borkdudecompared to clojure.spec, how slow?#2021-02-2115:44ikitommithe sequence schemas are at least an order of magnitude faster than spec in initial tests. Full function verification uses test.check, I think that dominates, and quess they are about the same.#2021-02-2115:47ikitommibut, having arity info available for fns at runtime (in clojure.core) would help to see that fns are correct on arity, without testing it and getting arityexceptions - costs a lot.#2021-02-2116:13borkdudeMaybe you can combine that with static analysis (e.g. from clj-kondo)?#2021-02-2118:09ikitommiI guess it could be done using a clj-kondo hook? with defaults, the test.check runs 19ms for a function on my machine. that’s 500 funcs per sec (on 1 core?). errors are faster, the given example fails fast on invalid return, under 1ms.#2021-02-2120:03borkdudeI mean:
> but, having arity info available for fns at runtime (in clojure.core) would help to see that fns are correct on arity, without testing it and getting arityexceptions - costs a lot.
You could use clj-kondo to find the correct arities perhaps#2021-02-2118:10ikitommii believe it could be made faster, less iterations on arguments etc.#2021-02-2118:11ikitommi[metosin/malli "0.3.0-SNAPSHOT"] has the stuff in.#2021-02-2119:59ikitommiwith 10 iterations, it drops to 0.3ms per fn.'#2021-02-2120:42emccue> e.g. not a type system.#2021-02-2120:42emccuei mean#2021-02-2120:43emccuefor me, it would serve much the same purpose i'd think#2021-02-2120:43emccueor rather - the same workflow#2021-02-2120:43emccuemaybe i'm missing a worldview though#2021-02-2121:07borkdudeBtw, this sounds interesting, but I haven't heard anything about it yet: https://www.fulcrologic.com/copilot#2021-02-2208:28ikitommilooking forward for that. I’m watching the guardrails repo, but think that’s going to be much more.#2021-02-2208:28ikitommi> (extensible, but starting with Clojure spec)#2021-02-2208:54borkdudeYeah, but will be a commercial product, so hard to contribute and see what's going on probably#2021-02-2211:06dharriganLooking forward to the new function schema - just saw the tweet!#2021-02-2211:53ikitommi@emccue Malli is a dynamic schema/type system, it’s not a static type system. tools like clj-kondo and typed clojure can do static analysis, maybe copilot too? Would love to see tooling get better and happy to help, but no time or skills to do anything non-trivial except to integrate into existing tooling. My point was that resolving fn arity using generative testing or 3rd party tools (clj-kondo) is a hard way to do something that would be easy to do in the core language itself. With core clojure: does a ring middleware chain would work for an async (3) arity? run and see if it throws arityexception 😞#2021-02-2211:55ikitommiI’m hope and think we’ll see great new developer tooling for clojure in 2021, from the community.#2021-02-2214:44ikitommiadded more benchmarks to repo, parsing vs spec:
;; 44µs
(let [spec (s/* (s/cat :prop string?,
:val (s/alt :s string?
:b boolean?)))
parse (partial s/conform spec)]
(cc/quick-bench
(parse ["-server" "foo" "-verbose" "-verbose" "-user" "joe"])))
;; 2.5µs
(let [schema [:* [:cat*
[:prop string?]
[:val [:alt*
[:s string?]
[:b boolean?]]]]]
parse (m/parser schema)]
(cc/quick-bench
(parse ["-server" "foo" "-verbose" "-verbose" "-user" "joe"])))#2021-02-2216:23Jakub ZikaHi guys,
I am trying to generate some random date with (mg/generate inst?)
but i am not sure how :seed or :size works here.
I am getting 1969 or 1970 in 99% cases. How to get a date like any >1980? Thanka
dwh-replicator.database> (mg/generate inst? {:seed 20})
;; => #inst "1970-01-01T00:00:00.000-00:00"
dwh-replicator.database> (mg/generate inst? {:seed 42})
;; => #inst "1970-01-01T00:00:00.000-00:00" #2021-02-2216:43juhoteperiI think in this case Malli just use spec.alpha generator for instants:
https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L178
https://github.com/clojure/spec.alpha/blob/31165fec69ff86129a1ada8b3f50864922dfc88a/src/main/clojure/clojure/spec/gen/alpha.clj#L160#2021-02-2216:55pithyless@jakub.zika-extern there's a good talk that explains how :seed and :size interact when writing custom generators. It even mentions the datetime problem @ 31:54. https://youtu.be/F4VZPxLZUdA?t=1911
1. The proposed solution in the talk is a custom generator that splits the datetime into separate domain bits (year / month / day / etc.)
2. A different solution (as seen e.g. https://github.com/nasa/Common-Metadata-Repository/blob/master/common-lib/src/cmr/common/test/test_check_ext.clj#L255-L257) is to change the way you choose an initial seed integer (that is used to coerce to date).#2021-02-2216:56Jakub ZikaThank you!#2021-02-2221:14Jakub ZikaI am mapping DB types to clojure types, so varchar(255) goes to [string? {:min 0 :max 255}] etc.
I am using these derived schemas to:
1. validate data : (malli/validate [string?] "hoy")
2. generate sample data : (malli.generator/generate [string?])
When I will go for custom generators then I will have to create new validators, correct?#2021-02-2221:19pithylessYou'll need to pass in the custom generator to the spec (not necessarily when validating, but at the very least wen generating sample data).#2021-02-2221:20pithylessso instead of
[string? {:min 0 :max 255}]
you'll have to wrap it, eg:
[:and {:gen/elements ["kikka" "kukka" "kakka"]} [string? {:min 0 :max 255}]]
#2021-02-2221:20pithylesswhere you use one of :gen/elements, :gen/gen, :gen/fmap, etc.#2021-02-2221:25pithyless^ I think you might even be able to just change the default :gen/gen of the specific type by modifying the malli registry globally (but then you're changing it for everything... which has its own issues ;))#2021-02-2221:26pithylessOR, perhaps make a custom registry where you override the :gen/gen for the basic types you're interested in (e.g. instant); and then pass in the custom registry only when generating sample data#2021-02-2306:26ikitommino need to wrap into :and, this works too: [:string {:min 0, :max 100, :gen/min 10, :gen/max 20, :gen/fmap (partial * 2)}]#2021-02-2306:27ikitommithe default schema generators are implemented as multimethods, so for global effects, one can just re-mount generator fof 'inst? for example. not recommended.#2021-02-2218:07Alex WhittWould anyone like to jump on this discussion thread I created on the subreddit? (I'd prefer to keep it there so it doesn't disappear behind Slack's paywall)
https://www.reddit.com/r/Clojure/comments/lpv8ok/spec_vs_malli/#2021-02-2318:12ikitommifinal change to figure out proper names for named branch -variants for: :cat, :alt and :or. they are currently :cat*, :alt* and :or* :face_with_rolling_eyes:
(m/parse
[:*
[:alt*
[:s :string]
[:i :int]]]
[1 "2" 3 "4"])
; => [[:i 1] [:s "2"] [:i 3] [:s "4]]#2021-02-2318:14ikitommi:catn, :altn & :orn?#2021-02-2318:14ikitommi:cat-named, :alt-named & :or-named?#2021-02-2318:15ikitommi:alley-cat, :danger-mouse & :skeletor?#2021-02-2407:28ordnungswidrig:cat-tag :alt-tag :or-tag ?#2021-02-2413:05Vincent Cantini suggest :named-cat, :named-alt, :named-or#2021-02-2413:23danielneal[:cat {:named? true}]?#2021-02-2814:09ikitommiusing properties would be nice for users, but would make the parser more complex. and updating properties might change the behaviour of the schema silently:
[:cat {:named? true} [:tuple int?]]#2021-02-2814:09ikitommi[:cat {:named? false} [:tuple int?]]
… is a totally diferent thing.#2021-02-2507:17steveb8nQ: can I use Malli inside a babashka script? (i.e. does it run inside sci?)#2021-02-2507:17steveb8nthis would be useful for validating tools.cli parsed options#2021-02-2507:43borkdudeNot yet, but feel free to post an issue, so we can gather community feedback. Meanwhile there are two options: spartan.spec and minimallist both work #2021-02-2508:32ikitommithere is https://github.com/metosin/malli/issues/302. Help most welcome#2021-02-2508:35borkdudeI think it will be quite a lot of work to port malli to bb compatible code. @ikitommi What is the API stability of malli? At one point we could add it to bb proper perhaps. I want to consider this, but also want some feedback from the "bb community" on this#2021-02-2508:37ikitommiVery stable. the named-schemas will change, but won’t change after release.#2021-02-2508:37ikitommipublic api has been immutable for 6+ months, no plans on breaking.#2021-02-2508:39steveb8nthanks guys. since it’s only tools.cli in CI (i.e. almost never changes), I can write a custom vaildator instead#2021-02-2508:40borkdude@U0510KXTU OK. This lib https://github.com/green-coder/minimallist also works with bb btw#2021-02-2508:42steveb8ncool. I’ll give it a try. thx#2021-02-2508:44steveb8nI like that it explicitely does not try to be fast. probably means they can deliver features faster#2021-02-2508:44steveb8nunlike Malli where I’m getting value out of the perf vs spec. perf is a feature#2021-02-2508:45borkdudeperf is a feature and the style of writing code (reifying Java classes) isn't well supported by bb from source for any Java class :)#2021-02-2508:46borkdudeWell, it is supported for records, etc, but definitely not fast#2021-02-2508:46ikitommimalli reifies just protocols, does that work ok?#2021-02-2508:46ikitommiis there something that we could do on malli side to make it work?#2021-02-2508:47steveb8nit’s all tradeoffs. I’m glad Tommi likes fast things 🙂#2021-02-2508:47borkdudeit does work:
$ bb -e '(defprotocol Foo) (instance? Foo (reify Foo))'
true
#2021-02-2508:48borkdude@ikitommi Maybe using reader conditionals helps, #?(:bb :foo :clj :bar)#2021-02-2508:48borkdudefor the parts that bb doesn't support#2021-02-2508:48ikitommisure, happy to add. which parts? 🙂#2021-02-2508:48borkdudetry and you will find out :) happy to comment on those parts#2021-02-2508:49ikitommithe protocol cache thing at least? (hacking over dead slow satisfies?)#2021-02-2508:49ikitommijust write a bb script of some malli code and see what breaks?#2021-02-2508:50borkdudeyes. in the malli repo, write a script like:
(require '[babashka.classpath :as cp])
(cp/add-classpath "src")
and then
(require '[malli.core :as m])
#2021-02-2508:50ikitommiexample/test-bed most welcome :)#2021-02-2508:51ikitommithat simple. cool.#2021-02-2508:51ikitommiwill add that to our next tech/hack-friday, which is… tomorrow.#2021-02-2508:53ikitommiso, does the :bb conditional work already?#2021-02-2508:53borkdude@ikitommi Actually, this is better:
(ns bb-script)
(require '[babashka.deps :as deps]
'[clojure.edn :as edn])
(deps/add-deps (edn/read-string (slurp "deps.edn")))
(require '[malli.core :as m])#2021-02-2508:53borkdude@ikitommi Yes, :bb is prioritized over :clj#2021-02-2508:53borkdudeit's order dependent#2021-02-2508:56borkdude@ikitommi So the first hack I did:
(ns malli.sci
#
#2021-02-2508:56borkdudeThe nest error:
----- Error --------------------------------------------------------------------
Type: java.lang.Exception
Message: Unable to resolve classname: java.util.ArrayDeque
Location: /private/tmp/malli/src/malli/impl/regex.cljc:35:3
----- Context ------------------------------------------------------------------
31: recognition for `validate`."
32:
33: (:refer-clojure :exclude [+ * repeat cat])
34: (:require [malli.impl.util :as miu])
35: #?(:clj (:import [java.util ArrayDeque])))
^--- Unable to resolve classname: java.util.ArrayDeque#2021-02-2508:56borkdudeSo we don't have that class in bb (yet)#2021-02-2508:57borkdudeI see that for cljs you didn't use it#2021-02-2508:58borkdudeI guess for bb you can take the :cljs branches there#2021-02-2508:59borkdudeThen after doing that, I run into:
(deftype ^:private CacheEntry [^long hash f ^long pos regs])
#2021-02-2509:00borkdudeThere are other deftypes in regex.cljc#2021-02-2509:00ikitommiwhat would be a good workaround for those?#2021-02-2509:00ikitommior, will bb support those at some point?#2021-02-2509:01ikitommigood to understand how to create bb-compatible code for the future.#2021-02-2509:01borkdudeSo these are the types of things that are not supported yet. Classes can be added, but deftype isn't supported.
I don't know what to do with deftype yet. Records are "faked" using normal maps + metadata.#2021-02-2509:04borkdudeI have seen a similar thing with meander vs matchete.
Matchete is written in a "simple" Clojure style so it just works with bb out of the box.
https://github.com/xapix-io/matchete
Where meander focuses on performance and this results in incompatible code.#2021-02-2509:05borkdudeMeander now has an "interpreter" which is compatible with bb which also more flexible than the macro style enforced in the main lib#2021-02-2512:13borkdudeIs there a way to get the humanized string for a schema, without any input?
(prn (me/humanize [:< 100]))
;;=> "should be smaller than 100"
#2021-02-2514:26ikitommioh, with nil value you mean?#2021-02-2514:32borkdudewith no value at all#2021-02-2514:32borkdudeit seems that the returned string doesn't depend on the input#2021-02-2514:19ikitommiyou can either:
[:int {:max 100}]
or:
[:< {:error/message "should be smaller than 100"} 100]#2021-02-2514:20ikitommithere are set of "type" schemas, which read properties: :int, :string , ...#2021-02-2514:22ikitommimight map all predicates schemas into these internally, which would make things like JSON Schema transformations & normal transformations simpler, need just to be defined for the base type, not to all predicates.#2021-02-2514:22ikitommipos-int? -> [:int {:min 1}]...#2021-02-2514:53borkdude@ikitommi What I mean is: you can get a string from humanize but this needs some input too which you first have to validate. However, the resulting string doesn't seem to be related to the input at all. This made me wonder: is it possible to get some "model" error message from a schema only, without validating anything#2021-02-2514:53borkdudeThis can then be passed to the second arg of the :validate tuple in tools.cli#2021-02-2514:54borkdudeNot that important, I can make the string itself manually or by validating some dummy input#2021-02-2516:09ikitommimalli supports both fixed error messages and functions generating error messages, the two examples here: https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L62-L68#2021-02-2516:10ikitommi... and messages can be localized. So, no simple way to do without value.#2021-02-2516:11ikitommithere are helpers in malli.error to pull out the error message/fn out of a schema, but value is needed for the fn case#2021-02-2516:14ikitommialso, one schema can emit many different errors, humanized by the error type. Maps emit extra-keys, missing-keys, missspelled-keys, etc.#2021-02-2610:49borkdudeDid this experiment. It brings malli to babashka via a pod. Due to pod limitations it might not bring you all the features, but might still be useful somehow. Feedback welcome 😊#2021-02-2619:57bartukanot sure if ask in #malli or #reitit channel, but nesting routes with path params are not merged correctly. I found this issue https://github.com/metosin/reitit/issues/462 there is any update on this?#2021-02-2620:05juhoteperiNo updates yet.
It is Reitit issue, Malli has malli.util/merge function that would work here, but Reitit doesn't have option to control how the values are merged, and because it is currently done using external library (https://github.com/weavejester/meta-merge) it is not simple to change.
Might have to copy meta-merge logic to Reitit and extend with controls to use custom function in certain paths.#2021-02-2620:56juhoteperihttps://github.com/metosin/reitit/pull/474 adds tests and found the place where we could call malli merge#2021-02-2620:05juhoteperiNo updates yet.
It is Reitit issue, Malli has malli.util/merge function that would work here, but Reitit doesn't have option to control how the values are merged, and because it is currently done using external library (https://github.com/weavejester/meta-merge) it is not simple to change.
Might have to copy meta-merge logic to Reitit and extend with controls to use custom function in certain paths.#2021-02-2620:56juhoteperihttps://github.com/metosin/reitit/pull/474 adds tests and found the place where we could call malli merge#2021-02-2620:58ikitommithere is also the nil-issue, that won't be fixed in meta-merge. Would simplify a lot how reitit route data can be used. metosin/ctrl-merge, right?#2021-02-2620:58ikitommihttps://github.com/weavejester/meta-merge/issues/11#2021-02-2814:06ikitomminaming is hard. to merge or not to merge: https://github.com/metosin/malli/pull/378#2021-03-0116:50ElsoDidn't really know where to put it, more a general metosin issue, but I'm getting
Syntax error macroexpanding at (core.clj:152:3).
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:581).
com.fasterxml.jackson.core.util.JacksonFeature
when loading metosin/jsonista 0.3.1#2021-03-0120:22mynomotoI'm trying the current master a58e04b265b1b6658bb9fe791fd103cfab452e53 for clj-kondo linting of function specs and it looks like that the configuration is generating the :ret key as shown on the readme. Was that removed on purpose, or am I doing something wrong?#2021-03-0120:27mynomotoI think https://github.com/metosin/malli/commit/0a22ccb1036c25b12f4b8c0d0f1dc20682aa6ca1 broke it, will create a pr.#2021-03-0201:41bartukaI am trying to use :* and got this message {:type :malli.core/invalid-schema, :data {:schema :*}} there is some setup to enable regex-like support?#2021-03-0201:51bartukaI think https://github.com/metosin/malli/blob/a58e04b265b1b6658bb9fe791fd103cfab452e53/src/malli/core.cljc#L1873 (sequence-schemas) happened after the last release hehe#2021-03-0202:30mynomotoIf you are using tools-deps you can reference a github commit.#2021-03-0208:12Yevgeni TsodikovHi all,
I’m trying to define a map with several optional fields, that at least 1 of them must exist.
For example, :a and :b are optional, :c is required.
{:a 1 :c 1} ; valid
{:b 1 :c 1} ; valid
{:a 1 :b 1 :c 1} ; valid
{:c 1} ; nope, invalid
As a reference, with `spec` , I’d write something like
(s/def ::my-map
(s/keys :req-un [::c (or ::a ::b)]))
#2021-03-0209:10ikitommi@evg.tso you can compose with :and, see example here: https://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D#2021-03-0209:39Yevgeni TsodikovThe thing with :and is that is doesn’t survive mu/merge very well.
To be more specific, my use case is with reitit.
There are several routes with common fields, but each route has specific fields that are different from one another.
• POST /share/this would accept either :a or :b and must contain :c (for example {:a 1 :c 1}
• POST /share/that would accept either :a or :b and must contain :d (for example {:a 1 :d 1}
.
└── schema with optional fields (at least one must exist)
├── mu/merge-ed schema with specific fields
└── mu/merge-ed schema with specific fields#2021-03-0209:41Yevgeni Tsodikov(mu/merge
[:map [:specific-field string?]]
[:and
[:map [:x int?] [:y int?]]])
=> [:and [:map [:x int?] [:y int?]]]
#2021-03-0209:44ikitommiI see. What if :and was considered as a constrained-kinda thing, where the first value is the actual schema and the rest are just extra rules for that. Would solve a lot of things. :thinking_face:#2021-03-0209:45ikitommineed to check would it break existing contracts or not.#2021-03-0209:47ikitommiwould this be more correct:
(mu/merge
[:map [:specific-field string?]]
[:and
[:map [:x int?] [:y int?]]
map?]])
=> [:and [:map [:spesific-field string?] [:x int?] [:y int?]] map?]
#2021-03-0209:47Yevgeni TsodikovThat would be great!#2021-03-0209:50Yevgeni TsodikovIn most of my schemas I use :and to add additional data like a :fn or a custom error message.#2021-03-0212:06ikitommi[metosin/malli "0.3.0"] 🥳 🥳 🥳#2021-03-0212:40delaguardoHi! I’m using malli together with reitit and reitit-swagger and I found a problem when I start using custom malli’s registry. tldr; generated swagger partials for an endpoint contains “definitions” key which is not valid according to swagger2 spec. also it messes up with references trying to point to one of defined schemas. As a workaround I added a middleware to walk throw generated object which will be encoded as json and pull all definitions to the top level but it looks ugly and not generic enough. Also there is a small inconsistency in encoding keywords to json.
https://malli.io/?value=%5B1%20%5B2%20%5B3%20%5B4%20nil%5D%5D%5D%5D&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%3Afoo%2FConsCell%20%5B%3Amaybe%20%5B%3Atuple%20%3Aint%20%5B%3Aref%20%3Afoo%2FConsCell%5D%5D%5D%7D%7D%0A%20%3Afoo%2FConsCell%5D
here is an illustration — :$ref key contains a string "#/definitions/:foo/ConsCell" with : inside. and there is a key :foo/ConsCell in :definitions after encoding to json this schema will be invalid because of : present in “$ref” key#2021-03-0213:02ikitommiI see. PR would be welcome to fix this. Maybe stringify all reference keys?#2021-03-0213:05delaguardoI would like to try ) where such fix should go? reitit-swagger or malli itself?#2021-03-0213:14ikitommiIn malli, either malli.json-schema or malli.swagger ns.#2021-03-0213:14delaguardocool, will be back with PR )#2021-03-0214:08stathissideriscongrats on the release!#2021-03-0413:24borkdudeNice tweet @mikethompson :) https://twitter.com/wazound/status/1367303612028751872#2021-03-0413:25borkdudeI hear similar sentiments in the latest #defnpodcast - there they called it the Osborne effect.
https://en.wikipedia.org/wiki/Osborne_effect#:~:text=The%20Osborne%20effect%20is%20a,announcing%20a%20future%20product%20prematurely.#2021-03-0413:30borkdudeI have the exact same problem with babashka: waiting for spec2 so not including spec1: and this can result in years of having nothing perhaps#2021-03-0413:31mikethompsonI didn't hear that podcast, but yeah, I feel like Clojure needs an outcome.
We're stuck in something of a twilight zone. And this is an important issue.
And right now Malli is just better.
Unless there is something pending with spec ... it feels like we need permission to get on with making Malli "it"#2021-03-0413:33borkdudeIn some way, no matter how good your library is, you can't compete with core since people will just believe that "core is better" no matter what you do, which isn't fair maybe.
But "better" is probably not the right way to describe it: spec and malli have different approaches. spec is more of an RDF approach where each unique attribute name describes a schema#2021-03-0413:34mikethompsonI'm the same as you: should re-frame support spec or Malli.
I've held off for a long time.#2021-03-0413:35borkdudeWhy should re-frame have to choose here? Can't this be decided in some add-on lib? Why do you need to address this concern in re-frame itself?#2021-03-0413:35mikethompsonI could#2021-03-0413:35borkdudeRe-frame events are already associated with globally namespaced keywords which maybe aligns well with spec#2021-03-0413:36mikethompsonBut if ever spec is ratified as the one true way for Clojure, I'd like for it to be the one true way for re-frame too#2021-03-0413:36mikethompsonAnyways#2021-03-0413:37borkdudeMaybe it can be made pluggable#2021-03-0413:45juhoteperiReitit supports all three so the application can choose.#2021-03-0413:45juhoteperiPluggable solution is probably also quite important in Cljs apps, because either Spec or Malli both add about 100KB JS to the output bundle (before gzip).#2021-03-0413:46juhoteperiI was surprised by this recently when I checked project output report in app that uses Malli, but re-chain uses Spec internally: https://github.com/ingesolvoll/re-chain/issues/6#2021-03-0415:38ikitommiInteresting discussion. About Malli cljs-size. The core has been developed DCE in mind, you can make a Malli bundle with just validation of strings, numbers, maps, vector and sets and it’s few kilos zipped. Currently, many extensions (humanized errors, generation, json schema etc) are implemented using multimethods, so pulling anything out of those, makes it much bigger.#2021-03-0415:39ikitommiThe Code Size Expression Problem.#2021-03-0415:40ikitommi#2021-03-0415:40ikitommihere’s the bare-bones malli.#2021-03-0415:42ikitommi@mikethompson what would the spec/malli integration look like? do you need something?#2021-03-0416:02juhoteperiAnd Reitit Malli coercion is built for Ring model, so it includes lots of features that are unnecessary for frontend routing. When we get around to implementing separate frontend coercion, it will drop dependency on some unused parts.#2021-03-0416:03juhoteperiLike the json schema generation parts.#2021-03-0416:12emccue^i'm not opposed to the json schema stuff but it does feel odd that it isn't a separate artifact#2021-03-0416:15juhoteperiIt is separate namespace on Malli.
On Reitit coercion impl. the parameter validation and generating the json-schemas for routes is closely related so they are on the same module currently.#2021-03-0416:17ikitommiyes, there should be a lite-version of reitit coercion, with just the encoding & decoding part - without throwing exceptions. Frontend would use that: simple, pure and small.#2021-03-0416:18ikitommimalli is designed so that malli.core is the essential ns, all others are optional (ok, there is nowadays malli.impl too). It has been easier to optimize the whole as everyhing is in single repo.#2021-03-0417:39arundilipanHi everyone, I’m not sure if this is a bug, but I’m running into a quirk with the validation and error reporting of :catn schemas#2021-03-0417:40arundilipanFor example if you had a schema like:
[:catn
[:amount [:fn {:error/fn '(fn [{:keys [value]} _] (str "Received value " value ", should be > 0"))
:decode/string malli.transform/-string->double}
'#(> % 0)]]
[:type [:enum "A" "B" "C"]]]#2021-03-0417:45arundilipanIf I try and (malli.core/explain schema [1.0 "D"]) , and humanize it, the error I get back is "Received value 1.0, should be > 0" , rather than "should be one of "A", "B", or "C"#2021-03-0417:47borkdudeIsn't the idea of :catn that you provide names for the schema elements?#2021-03-0417:47borkdudeElse you should just use :cat#2021-03-0417:56arundilipanFixed the example, I do intend to use :catn to provide names for the schema elements#2021-03-0420:00ikitommi@arundilipan seems to work:
(-> [:catn
[:amount [:fn {:error/fn '(fn [{:keys [value]} _] (str "Received value " value ", should be > 0"))
:decode/string malli.transform/-string->double}
'#(> % 0)]]
[:type [:enum "A" "B" "C"]]]
(m/explain [1.0 "D"])
(malli.error/humanize))
; => [nil ["should be either A, B or C"]]#2021-03-0420:11borkdudeshould :catn not give back error messages by name?#2021-03-0420:18ikitommierror messages by name?#2021-03-0420:19borkdude{:amount nil :type ["should be ..."]}#2021-03-0420:19arundilipanThat’s for :map i think#2021-03-0420:19borkdudeare the positions in :cat always unambiguous? I know for spec they are certainly not with s/cat#2021-03-0420:19ikitommihmm. not sure. the current humanize just mimics the raw result, not parsed one.#2021-03-0420:20ikitommibut the info is there:#2021-03-0420:20ikitommi(-> [:catn
[:amount [:double {:min 0}]]
[:type [:enum "A" "B" "C"]]]
(m/explain [1.0 "D"]))
;{:schema [:catn [:amount [:double {:min 0}]] [:type [:enum "A" "B" "C"]]],
; :value [1.0 "D"],
; :errors (#Error{:path [:type 0], :in [1], :schema [:enum "A" "B" "C"], :value "D"})}#2021-03-0420:21ikitommie.g. it’s a sequence, errors are positioned by :in.#2021-03-0420:21borkdudemaybe for tools like expound it would be nice to have the humanized error along with the path?#2021-03-0420:21borkdudeah I see#2021-03-0420:21borkdudeso using the explain output you can get to the error message by path#2021-03-0420:21borkdudeusing get-in or so#2021-03-0420:23ikitommifor map, the explain info is:
(-> [:map
[:amount [:double {:min 0}]]
[:type [:enum "A" "B" "C"]]]
(m/explain {:amount 1.0
:type "D"}))
;{:schema [:map [:amount [:double {:min 0}]] [:type [:enum "A" "B" "C"]]],
; :value {:amount 1.0, :type "D"},
; :errors (#Error{:path [:type 0], :in [:type], :schema [:enum "A" "B" "C"], :value "D"})}#2021-03-0420:24ikitommibut yes, the branch name is available in the error data for :catn, to be printed nicely etc.#2021-03-0420:26ikitommi@arundilipan merged a humanized error for :double, so this works now too:
(-> [:catn
[:amount [:double {:min 0}]]
[:type [:enum "A" "B" "C"]]]
(m/explain [1.0 "D"])
(malli.error/humanize))
; => => [nil ["should be either A, B or C"]] #2021-03-0421:10borkdudeuser=> (require '[malli.error :as e])
nil
user=> (require '[malli.core :as m])
nil
user=> (e/humanize (m/explain [:cat int? int? [:? int?] [:? string?]] [1 2 :foo]))
[nil nil ["should be an int" "should be a string" "unknown error"]]
Looks legit, except maybe for the "unknown error"?#2021-03-0421:13borkdudeThat's probably better expressed as:
(m/explain [:cat int? int? [:orn [:x int?] [:y string?]]] [1 2 :foo])
Just wanted to see what malli would make of it#2021-03-0422:04ikitommigood catch @borkdude, the regex error types didn’t have humanized forms. fixed in master:
(-> [:cat int? int?]
(m/explain [1])
(me/humanize))
; => [nil ["end of input"]]
(-> [:cat int? int?]
(m/explain [1 2 3])
(me/humanize))
; => [nil nil ["input remaining"]]
(-> [:cat int? int? [:? int?] [:? string?]]
(m/explain [1 2 :foo])
(me/humanize))
; => [nil nil ["should be an int" "should be a string" "input remaining"]]#2021-03-0422:05borkdudenice!#2021-03-0422:08ikitommiwith :or:
(-> [:cat int? int? [:or int? string?]]
(m/explain [1 2 :foo])
(me/humanize))
; => [nil nil ["should be an int" "should be a string"]]#2021-03-0422:14borkdudeyeah, that already worked right?#2021-03-0504:57bedersquick question with regards to properties.
I’m trying to put some properties into my schemas, but can’t seem to be getting them back.
What am I doing wrong here?
; malli 0.3.0:
; [malli.registry :as registry]
(def registry
(registry/composite-registry
malli/default-registry
{:common/single-line [:re {:bubu :lala} #"^[^\r\n]*$"]}))
(malli/properties :common/single-line {:registry registry})
=> nil
Expected: {:bubu :lala}
#2021-03-0507:34ikitommi@beders oh, that’s not good. registry interally wraps the registered schema instances into :malli.core/schema , which is an eager reference type. When you pull out an instance schema from a registry, you get the reference back. it mostly a pass-through, e.g. calling -validatorto the reference return the validator of the referenced schema. But: for some reason, the current impl returns the reference properties and options if asked. I think it’s a bad feature, should be changed.#2021-03-0507:36ikitommineed to think how that effects other things. before that, you m/deref safely:#2021-03-0507:37ikitommi(def registry
(mr/composite-registry
m/default-registry
{:common/single-line [:re {:bubu :lala} #"^[^\r\n]*$"]}))
(-> (m/schema :common/single-line {:registry registry})
(doto prn)
(m/deref)
(doto prn)
(m/properties))
;:common/single-line
;[:re {:bubu :lala} #"^[^\r\n]*$"]
;=> {:bubu :lala}#2021-03-0507:38ikitommiworkaround for now:
(def registry
(mr/composite-registry
m/default-registry
{:common/single-line [:re {:bubu :lala} #"^[^\r\n]*$"]}))
(defn schema [?schema]
(m/deref (m/schema ?schema {:registry registry})))
(schema :common/single-line)
; => [:re {:bubu :lala} #"^[^\r\n]*$"]#2021-03-0917:49bedersThanks for the help and explanation. It would be good to have some documentation around explaining instances vs. Schema. I’m still confused 🙂#2021-03-0507:41ikitommicomments welcome on how it should work.#2021-03-0508:25ordnungswidrigIs there a standard body of localized error messages for humanize?#2021-03-0508:28dharriganOnly english https://github.com/metosin/malli/blob/master/src/malli/error.cljc#2021-03-0508:28dharriganHowever, the docs do show how to add in other i18n messages#2021-03-0508:29dharriganhttps://github.com/metosin/malli#custom-error-messages#2021-03-0511:51borkdude@ikitommi What about making a plugin for malli which inspects clojure.spec specs and emits a malli schema from it? ;) Might help people migrating to malli#2021-03-0512:55ikitommi@borkdude brilliant idea. Let's do it.#2021-03-0517:12dviramontesHi, trying to wrap my mind around how these two examples can/will differ over time when it comes to value generation
(def CDN1
[:map
[:images [:vector string?]]])
(def CDN2
[:map
[:images [:sequential string?]]])
(malli.generator/generate CDN1)
(malli.generator/generate CDN2)
-- i guess the question is - will they defer ? and if so how ? in my testing the generated values are very similar#2021-03-0517:15borkdude@dviramontes in clojure sequential can also be a list or lazy-seq for example, not always a vector, but that should not affect equality semantics#2021-03-0517:17borkdudeit seems the generator doesn't really generate anything other than vectors at the moment (by experimentation) but it could#2021-03-0517:18dviramontesgotcha, thanks very much!#2021-03-0517:19borkdudeTIL, malli also supports keywords as predicates:
(malli.generator/generate [:sequential :int])
I prefer that syntax personally#2021-03-0517:20borkdudeuser=> (malli.core/validate :int 1)
true
#2021-03-0620:12ikitommipulling out malli schemas for defns (Vars):
(-var-schema #'+)
;[:function
; [:=> :catn :any]
; [:=> [:catn [x :any]] :any]
; [:=> [:catn [x :any] [y :any]] :any]
; [:=> [:catn [x :any] [y :any] [more [:+ :any]]] :any]]
(-var-schema #'println)
; [:=> [:catn [more [:* :any]]] :any]
#2021-03-0620:17borkdudeIs this valid syntax?
[:=> :catn :any]
#2021-03-0620:19ikitommiyes, vectors are optional if there are no childs or props, e.g. [:int] can be written as :int.#2021-03-0620:20ikitommisame for :catn. “empty sequence with named children”#2021-03-0620:20ikitommithe imp btw:
(defn -var-schema [var]
(let [-=> (fn [as] (let [[f s [t]] (partition-by #{'&} as)
[fas ras rop] (cond t [f t :+], s [nil (first s) :*], :else [f])]
[:=> (if (or (seq fas) ras)
(vec (concat [:catn] (mapv (fn [a] [a :any]) fas) (when ras [[ras [rop :any]]])))
:catn) :any]))
{:keys [arglists]} (meta var)
-=>s (mapv -=> arglists)]
(if (second arglists) (into [:function] -=>s) (first -=>s))))#2021-03-0620:21ikitomminot sure if it’s a good idea to use :arglistsvar meta, but, works on my repl just now at least 🙂#2021-03-0620:28borkdudenot a bad idea#2021-03-0620:29borkdudesome Clojure vars use something like x* which denotes multiple xs, so the arglist is not always reliable, but in most cases it's auto-generated from the defn itself#2021-03-0712:02mike_ananev@ikitommi what is the difference between :title and :description in spec? I want to describe my config file using malli and guessing what should I use to describe every field in the config map.#2021-03-0712:09mike_ananev(def http-server-host [:and {:title "http server host"} ne-string])
;; or
(def http-server-host [:and {:description "http server host"} ne-string])#2021-03-0712:24ikitommi@mike1452 those are pulled from JSON Schema convention:
> The `title` and `description` keywords must be strings. A “title” will preferably be short, whereas a “description” will provide a more lengthy explanation about the purpose of the data described by the schema.
I would use :description there. In OpenAPI & Swagger, title is pulled as the Schema name, e.g.
[:map {:title "User", :description "Describes User of the System"}
[:name {:description "Name of the User"} :string]
[:age {:description "Age, must be >= 18} [:int {:min 18}]]]#2021-03-0712:28mike_ananevThank you.#2021-03-0712:43ikitommipulling the return types too:
(-var-schema #'str)
;[:function
; [:=> :catn :string]
; [:=> [:catn [x :any]] :string]
; [:=> [:catn [x :any] [ys [:+ :any]]] :string]]
(-var-schema #'distinct?)
;[:function
; [:=> [:catn [x :any]] :boolean]
; [:=> [:catn [x :any] [y :any]] :boolean]
; [:=> [:catn [x :any] [y :any] [more [:+ :any]]] :boolean]]
#2021-03-0712:53borkdudeClever:
user=> (meta (second (:arglists (meta #'str))))
{:tag java.lang.String}#2021-03-0713:01ikitommiNoticed too that the hand-written arglists seem to have weird syntax.#2021-03-0716:06joshkhloving malli so far! what's the idiomatic way of "extending" a schema? for example let's say i have a Person map with the basic attributes first-name and last-name, and then a ProfessionalPerson that is a Person with an additional required profession attribute. would i create a ProfessionalPerson definition that contains only just [:map [:profession string]] and then mu/merge it on top of the Person map [:map [:first-name string?][:last-name-string?]]?#2021-03-0716:12borkdude@joshkh Or mu/assoc#2021-03-0716:17joshkhcool, thanks borkdude. on a side note it's too bad that mu/assoc doesn't allow for & kvs like clojure.core/assoc 😄#2021-03-0716:19borkdude;)#2021-03-0716:22juhoteperiMight be possible to implement. The options parameter makes it a bit inconvenient though, we could check that if odd number of params, last param is the options, and for even number of params, no options, just kv pairs. Not sure if worth the complexity.#2021-03-0716:23juhoteperimalli.util/dissoc also takes just one key.#2021-03-0716:23borkdudeif you need more than one k/v, you can just use mu/merge :)#2021-03-0716:24borkdudeor repeat mu/assoc#2021-03-0716:24joshkhyeah, merge is fine by me. i just thought it was funny that my first instinct was to treat it like the assoc i know.#2021-03-0716:26juhoteperiYeah.
Though I think it will be worth to mention the differences to clojure.core functions in the docstrings, now the util fns mention just "like clojure.core/x" but then many of them take a bit different parameters.#2021-03-0716:31joshkhperhaps. admittedly i just tried it out while completely ignoring the very clear docstring params in my editor. you can lead a horse to water but you can't make it drink 😉#2021-03-0716:33borkdudeIt's awesome that you can do this with malli btw.#2021-03-0716:37joshkhabsolutely. and the deep merging is really handy, too#2021-03-0717:34ikitommithere are at least two options to resolve the malli.util varargs issue:
1. make all utilities only work with Schema instances, e.g. no auto-coercion from schema AST => only place to pass the options would be the m/schema -> simpler, more boilerplate,
2. make a custom type/record/protocol for the options, could be the first argument in all functions, “the schema context” - easy to distinguish it from other args #2021-03-0719:47schmeedo I need to have sci for a spec like this?
[:map {:gen/fmap 'map->Point}
[:lat [:double {:min -180.0 :max 180}]]
[:long [:double {:min -180.0 :max 180}]]]]#2021-03-0719:53schmeeseems like it according to the tests: https://github.com/metosin/malli/blob/master/test/malli/generator_test.cljc#L94-L105#2021-03-0719:53schmeecan you configure malli to use the clojure compiler for eval instead of sci?#2021-03-0720:06schmeeI can’t get this to work with or without sci, can you not use namespace-qualified functions as arguments to :gen/fmap? :thinking_face:#2021-03-0720:07ikitommiyes, just don't quote and it works#2021-03-0720:07ikitommi(but, can't be deserialized if it's a fn value)#2021-03-0720:08ikitommiIf you wan't eval, PR welcome. Could be option :malli.core/evaluator a, there could be a -eval-evaualuator in malli.core#2021-03-0720:09ikitommirelevant code: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1739-L1741#2021-03-0720:12schmeeahh, I was fooled by the output of my REPL, it printed the record as a plain map (and skipped the record type)!#2021-03-0720:13schmeethen all is well, thanks for the help 🙂#2021-03-0810:03borkdudeuser=> (time (m/validate [:cat [:+ [:+ [:enum 0]]] [:enum 1]] [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]) )
"Elapsed time: 0.582137 msecs"
true
user=> (time (s/valid? (s/cat :zeroes (s/+ (s/+ #{0})) :one #{1}) [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]))
"Elapsed time: 1265.3679 msecs"
(got the example from https://quanttype.net/posts/2021-03-06-clojure-spec-and-untrusted-input.html)#2021-03-0810:17borkdudeAre there any examples of regexes that are slow in malli too?#2021-03-0810:24miikkaMalli's regular expression matching algorithm is fundamentally different, so as far as i know, it does not have the exponential slow down problem. Probably you could still come up with some slow examples, but I don't have insight into what is slow and what is fast.#2021-03-0810:32borkdudeBecause it doesn't have backtracking, right?#2021-03-0813:12miikkaIt does use backtracking, but see the ns docstring for discussion: https://github.com/metosin/malli/blob/master/src/malli/impl/regex.cljc#2021-03-0814:48borkdude#2021-03-0815:23arundilipanSorry for the late response @borkdude @ikitommi, I had an health emergency this week and so I wasn’t able to do anything about it#2021-03-0815:23arundilipanI did follow up with the example and it did work#2021-03-0815:23arundilipanI have a follow-up question, though#2021-03-0815:35arundilipanI’m wondering if m/decode with the above schema is supposed to produce that result, and if so, I was wondering how to solve this so that I return [1.0 "A"] from decode?#2021-03-0817:42ikitommi@arundilipan works on my machine:
(ns sample.core
(:require [malli.core :as m]
[malli.transform :as mt]))
(def GreaterThanZero
[:fn {:decode/string mt/-string->double}
'#(> % 0)])
(def TestEnum
[:enum "A" "B"])
(m/decode
[:catn
[:amount GreaterThanZero]
[:layout TestEnum]]
["1.0" "A"]
mt/string-transformer)
; => [1.0 "A"]#2021-03-0817:45ikitommiif the result doesn’t match the schema after the transformation, the top-level schema returns the original. this is a feature of the current regex impl, doesn’t support partial transformation:
(m/decode
[:catn
[:amount GreaterThanZero]
[:layout TestEnum]]
["1.0" "C"]
mt/string-transformer)
; => ["1.0" "C"]
#2021-03-0817:46arundilipanOh that’s it, you’re right#2021-03-0817:47arundilipanIn that case I guess the thing to do would be do use something like edn/read-string instead of decode? I’d like to make sure that the number is parsed regardless of the result of the rest of the :catn #2021-03-0817:48ikitommior, you could use non-regex schema like :tuple:
(m/decode
[:tuple GreaterThanZero TestEnum]
["1.0" "C"]
mt/string-transformer)
; => [1.0 "C"]
#2021-03-0817:49ikitommino :tuplen atm 😉#2021-03-0817:50ikitommiyou could write issue about the partial decoding with regex?#2021-03-0817:50ikitommi(might be easy to fix)#2021-03-0817:51arundilipanYea i’ll make an issue on the repo#2021-03-0817:51arundilipanRight now I’m trying to use the :catn variants because of the coercion into a map#2021-03-0817:55arundilipanDone!#2021-03-0918:06raymcdermottI'm trying to take a simple map like this an make it consumable by reitit-swagger#2021-03-0918:07raymcdermott(def Org
[:map
[:id Id]
[:ref string?]])
#2021-03-0918:08raymcdermottafter swagger/transform I get#2021-03-0918:08raymcdermott{:type "object",
:properties {:id {:type "string",
:pattern #"^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$"},
:ref {:type "string"}},
:required [:id :ref]}#2021-03-0918:11raymcdermottnot sure how / where to plug this in to the reitit.swagger structure#2021-03-0918:12raymcdermottlooking at the example in reitit I see this#2021-03-0918:12raymcdermott["/plus"
{:get {:summary "plus with malli query parameters"
:parameters {:query [:map [:x int?] [:y int?]]}
:responses {200 {:body [:map [:total int?]]}}
:handler (fn [{{{:keys [x y]} :query} :parameters}]
{:status 200
:body {:total (+ x y)}})}
:post {:summary "plus with malli body parameters"
:parameters {:body [:map [:x int?] [:y int?]]}
:responses {200 {:body [:map [:total int?]]}}
:handler (fn [{{{:keys [x y]} :body} :parameters}]
{:status 200
:body {:total (+ x y)}})}}]#2021-03-0918:14raymcdermottI can see that I should decorate the Org with some swagger but still, I'm being dumb cos I can't see how to smash everything together#2021-03-0918:15raymcdermottany clues if you have done this already would be appreciated#2021-03-0918:48ikitommi@raymcdermott if the swgger-transform output is ok, just use it like {:parameters {:body Org}} and it works. swagger-transformation is done in the reitit.swagger/create-swagger-handler code for all parameters & response schemas automatically, it looks the effective :coercion for a route and asks it to transform the stuff.#2021-03-0918:49ikitommihere’s the code: https://github.com/metosin/reitit/blob/master/modules/reitit-swagger/src/reitit/swagger.cljc#L69-L109#2021-03-0918:49ikitommiyou can add stuff with :swagger namespace or key:
(def Org
[:map {:title "Org"}
[:id {:swagger/description "id"} Id]
[:ref {:swagger {:default "kikka"}} string?]])#2021-03-0919:31raymcdermottok - great, I'll give a go. Thanks @ikitommi#2021-03-0919:51raymcdermottI can report success 🙏:skin-tone-3:#2021-03-1011:11Adam HelinsI have written a WebAssembly decompiler/compiler and I would like to "spec" the WASM intermediary representation I am using. After a long while of avoiding clojure.spec, I've tried using it for that purpose but it's just too hard, too limited. So I am trying out Malli. Given how flexible it is, I have high hopes. But I am a Malli noob so here I am.
My problem is actually a common one: a data schema were parts of the schema are references ("pointers") to other parts of the schema. Validation is easy but generation is hard. Spec makes that extra hard by forcing a global registry.
Let's stick to this WASM example to have something concrete to work with. There exists a type section (map of type index to type) and a function section (map of function index to function where a function holds an existing type index). Generating a valid representation in pseudo-code would look like:
1. Generate a type section where type index is a nat?
2. Mutate registry so that type index becomes an enum of generated type index
3. Generate a function section , now assured that generated type index exist
Solution A. I guess one way would be to do exactly that. Generating things in the right order, step by step, and keep on doing that while successively building a local registry based on what has been previously generated. I am unsure how practical that would be for a more complex example.
Solution B. Another way would be to have one schema with a "local mutable registry" and declaring custom generators along the way which would mutate this "local mutable registry" based on what they generate. However I understand that local registries must be concrete maps and do not allow this. On the longer term, I am not sure this would be more pratical than Solution A and it certainly is not very functional.
Any better, idiomatic way?#2021-03-1011:27Adam HelinsNote regarding Solution B: Unless I am doing it wrong, I cannot manage to use a mutable registry in a local way (ie. using it as an explicit :registry argument throws :malli.core/invalid-schema {:schema :map})#2021-03-1013:53ikitommi@adam678 Sounds really interesting! I can’t recall why the local registries only support maps. Would be most likely a small change it to support the Registry#2021-03-1013:55ikitommione option woud be to just collect stuff to a mutable registry or an custom atom, use it via (m/schema x {:registry registry}). when everything is collected and if the schemas are serializable, you could writen them into one local (immutable) registry.#2021-03-1014:14nilernThere was a similar PR recently https://github.com/metosin/malli/pull/337#2021-03-1014:29jjttjjI'm wondering what the "limits" of the malli value transformations should be philosophically. If I'm working with an external api that represents dates as strings, but I want to represent them as Instants in my app, is that a valid use for transformers? What about if the external api provides a nested "address" map that I want to eventually turn into an Address record type? It wouldn't necessarily be used as a two way transformation very much, I wouldn't necessarily be sending my addresses back to their api and need to encode them again as strings.
Is this just a separate problem, and I should just use a function to get things into my domain records/types?#2021-03-1014:30juhoteperiCoercing stings to proper types at least is fine use. Similar to coercing json or path or query string parameters to booleans, dates etc.#2021-03-1014:31nilernWe do those sorts of things all the time, also with Schema and Spec#2021-03-1014:34jjttjjSo it's good to use malli to get from string all the way to our full on domain objects?#2021-03-1014:38ikitommisure. If it like looks complex, then move the transformation out. I haven't seen a always valid limit. string->map map->record both perfect cases.#2021-03-1014:39jjttjjby "move the transformation out" you mean separate it into a different transformer?#2021-03-1014:40ikitommiwrote a while back this generic nonsense (on spec):
> Domain-specific data-macros are cool, but add some complexity due to the inversion of control. For just this reason, we have pulled out Schema-based domain coercions from some of our client projects. Use them wisely.#2021-03-1014:40ikitommihttps://www.metosin.fi/blog/clojure-spec-as-a-runtime-transformation-engine/#data-macros#2021-03-1014:40jjttjjThanks!#2021-03-1014:42ikitommiby moving out I meant into a separate function outside of schemas, e.g. (external->internal data) thing.#2021-03-1014:45jjttjjAnd then do you call that from a transformer still? or do you mean the data goes from
json-string-> clojure data -> malli transformers -> external->internal
#2021-03-1015:42ikitommi• optimal: json-string + malli -> internal
• current good practise: json-string -> EDN -> malli transformers -> internal
• if the internal->external is complex. uses external data etc: json-string -> EDN -> malli transformers -> external->internal#2021-03-1015:44ikitommi(did a spike on deriving jackson-decoder from malli schema, but nothing production grade atm, in theory, shoud be much faster)#2021-03-1015:51jjttjjthat makes sense, thanks again!#2021-03-1015:09Adam Helins@ikitommi All right, thanks, I got a sense of where to start. As a beginner I was mostly troubled by the fact that local registries can be only map-based. Since you are not sure this is intended, I took the liberty of opening an issue: https://github.com/metosin/malli/issues/389#2021-03-1018:36EdHi. If I have a :dispatch multi-schema, is there an easy way to put a catch-all else clause in the matches?#2021-03-1019:00ikitommi@l0st3d not atm, but could, does [:or [:multi …] :default]work for you?#2021-03-1019:00ikitommimaybe there could be a :malli/default branch?#2021-03-1019:27EdI think I've worked out how to write what I was trying to write in a different way (using :or, but I needed an exclusionary condition instead of :default), but it might be a useful feature to add ... I quite like the idea of :malli/default. Partly because of the parallel to multimethods. Maybe there's a clean way of overriding the keyword?#2021-03-1019:32Edmy exclusion clause in the second branch of the or is a bit big, but cos it's all data it's ok to write a function to produce the branches 😉#2021-03-1105:14ikitommi@l0st3d & everyone else, opinions on :m/default or :malli/default?
[:multi {:dispatch :type}
["object" [:map-of :keyword :string]]
[:m/default :string]]
vs:
[:multi {:dispatch :type}
["object" [:map-of :keyword :string]]
[:malli/default :string]]
… same could be used for extra-keys in maps, e.g. :m/extra vs :malli/extra. Personally fond of the :m.#2021-03-1105:14ikitommihttps://github.com/metosin/malli/pull/391#2021-03-1105:15ikitommi(def valid?
(m/validator
[:multi {:dispatch :type}
["object" [:map-of :keyword :string]]
[:m/default :string]]))
(valid? {:type "object", :key "1", :value "100"})
; => true
(valid? "SUCCESS!")
; => true
(valid? :failure)
; => false#2021-03-1108:37EdThat would definitely solve my problem and make my code easier to read ... it get's my vote ... many thanks for the impressively fast feedback.#2021-03-1108:41EdAlthough maybe ::m/default might be better#2021-03-1108:41EdAh ... I see someone has already commented to that effect on the pr 😉#2021-03-1110:40Adam HelinsHehe, sometimes I dream about creating a sub-community of "Clojurists against too much abbreviation"#2021-03-1110:48Ed😉 ... I think it's not about abbreviation, but scope ... :m/ is a public namespace that may be being used by an application and may well be a target in that multi dispatch, whereas ::m/ is scoped to malli.core and therefore shouldn't have meaning in your application - so it's fine to use as a default ... I think where clojure pushes back against brevity it tends to be for reasons of applicability in different contexts - which makes it a more generally useful language#2021-03-1114:33armedHello, how to attach custom error message to seqexp? For example:
(malli.error/humanize
(malli/explain
[:map
[:items
[:+
{:error/message "Items must not be empty"}
[:map [:foo string?
:bar int?]]]]]
{:items []}))#2021-03-1117:46shanlooks like the :error/message is in the wrong place:
(me/humanize
(m/explain
[:map
[:items
[:+
[:map {:error/message "Items must not be empty"}
[:foo string?
:bar int?]]]]]
{:items []}));; => {:items [["Items must not be empty"]]}#2021-03-1118:55armedThanks, thats interesting. My code works if I replace :+ with :vector#2021-03-1118:57armedAnd if I put error inside :map, then all other map validation errors replaced with that one, even ‘missing key’ is becomes ‘items must not be empty’. #2021-03-1114:34armedI get {:items [["unknown error"]]} error#2021-03-1117:59emccueHow would I represent values like this#2021-03-1117:59emccue:keyword-a
:keyword-b
{:map "with" :some 'shape}#2021-03-1118:00emccue(in the same schema)#2021-03-1118:02emccueat least with the online tool, alt doesn't seem to do what I would expect#2021-03-1118:03emccue#2021-03-1118:33ikitommi@emccue try https://malli.io/?value=%5B%3Aa%20%3Ab%20%7B%3Aname%20%22kikka%22%7D%5D&schema=%5B%3Avector%20%0A%20%5B%3Aor%20%0A%20%20%5B%3Aenum%20%3Aa%20%3Ab%5D%0A%20%20%5B%3Amap%20%5B%3Aname%20%3Astring%5D%5D%5D%5D#2021-03-1118:34ikitommithe http://malli.io is not updated for 0.3.0, so no :alt there yet. PR welcome to update deps#2021-03-1118:38emccuewould alt work in that case?#2021-03-1118:43nilern:alt is for sequences only#2021-03-1118:47nilern[:alt [:= :a] [:= :b]] matches [:a] and (:b) but not :a#2021-03-1119:08emccueinteresting that [:enum :a :b] is different than [:or [:= :a] [:= :b]]#2021-03-1119:23ikitommidifferent, how?#2021-03-1120:06emccuelike, they are inferred differently#2021-03-1200:42emccue(looped back to this, i'm probably wrong and got confused by the "inferred schema" part and also my bad memory)#2021-03-1119:19ikitommiupdated http://malli.io, with few sequence samples (args & hiccup)#2021-03-1119:20ikitommihttps://malli.io/?value=%5B%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D%5D&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D#2021-03-1119:20ikitommihttps://malli.io/?value=%5B%22-server%22%20%22foo%22%20%22-verbose%22%20true%20%22-user%22%20%22joe%22%5D&schema=%5B%3A*%20%5B%3Acatn%20%5B%3Aprop%20string%3F%5D%20%5B%3Aval%20%5B%3Aaltn%20%5B%3As%20string%3F%5D%20%5B%3Ab%20boolean%3F%5D%5D%5D%5D%5D#2021-03-1119:21ikitommithe Hiccup in JSON Schema looks fishy:
{:$ref "#/definitions/hiccup", :definitions {"hiccup" {}}}#2021-03-1119:22ikitommibut, there are no sequence schemas there, so i guess it’s the best we can do.#2021-03-1212:59raymcdermottI want to protect the swagger doc built from the malli data model using buddy. I have seen the reitit examples which protect routes but it's not clear how this integrates into the Swagger model ... I will also x post to reitit#2021-03-1215:52raymcdermott[ thanks for the answer by @ikitommi in #reitit ]#2021-03-1218:39ikitommimany ways of stripping extra keys from maps: https://www.reddit.com/r/Clojure/comments/m3hx1e/how_do_i_walk_a_complex_map_and_remove_keys_based/#2021-03-1221:36borkdudeJust had a thought. Can malli be used in places where currently meander or matchete (https://github.com/xapix-io/matchete) are used?
Although this works, it is a bit cumbersome:
user=> (m/validate [:cat [:enum 1] :any [:enum 3]] '[1 2 3])
true
#2021-03-1221:39borkdudeAnd how could I get the destructured value, after validation?
user=> (m/explain [:catn [:a [:enum 1]] [:b :any] [:c [:enum 3]]] '[1 2 3])
nil
#2021-03-1222:29borkdudeoh of course, parse facepalm
user=> (m/parse [:catn [:a [:enum 1]] [:b :any] [:c [:enum 3]]] '[1 2 3])
{:a 1, :b 2, :c 3}#2021-03-1222:49borkdudeIs there a way to indicate that you want to ignore a certain binding in the parsed output?#2021-03-1307:44ikitomminot at the moment. issue welcome.#2021-03-1222:57borkdudeI hacked it like this:
https://gist.github.com/borkdude/26906ee15585ed5e1b7a8eda4cc1ee18#2021-03-1307:45ikitommi::m/default for :multi merged in master:
(def valid?
(m/validator
[:multi {:dispatch :type}
["object" [:map-of :keyword :string]]
[::m/default :string]]))
(valid? {:type "object", :key "1", :value "100"})
; => true
(valid? "SUCCESS!")
; => true
(valid? :failure)
; => false#2021-03-1307:50ikitommiMaybe the same could be used for extra keys in maps? (https://github.com/metosin/malli/issues/43)#2021-03-1309:53borkdude@ikitommi I was trying (for fun) to write a core.match like thing with malli.
So {:a ?x :b 1} would match on {:a 2 :b 1} and parsing would give you back {?x 1}. This is currently a bit difficult since you cannot change how keys are named in the parsed output from map schemas, can you?#2021-03-1309:54borkdudeSo I would generate a schema like {:b [:= 1] :a :any} but then the parsed output loses the name ?x#2021-03-1310:07borkdudeCan I influence how things get parsed using an extra predicate similar to s/and in spec?
(m/parse [:map [:a [:and :any
[:fn (fn [x]
['x? x])]]]]
{:a 1})
#2021-03-1310:13ikitomminot atm, parsing could have it's own property key for this, e.g. [:map {:parse ...} ...]. Internally, could be interceptors, so one can do easily pre, post & schema-based parsing with that.#2021-03-1310:13ikitommiWe discussed with @nilern about combining internally parsing, explain and transform. They are now mostly (optimized) duplicates of each other.#2021-03-1310:15ikitommimap keys -> could done with same mechanism. Add custom parse tags to keys as properties and hook a post-parse fn to rename the keys. Or do, whatever.#2021-03-1311:04ikitommiported the plumatic-style inline schemas for 0.3.0, it kinda works, but no tests and not happy with the original (string-based) error reporting. renamed to ns to malli.experimental.schema as it might not be part of the malli core library. any thoughts on this? add tests, cleanup and ship as experimental add-on?#2021-03-1311:04ikitommihttps://github.com/metosin/malli/pull/305#2021-03-1311:06ikitommithe defn now emits a function schema into malli function registry and it can be configured to validate always, never or based on a dynamic var - at runtime.#2021-03-1414:18vemvReplied in https://github.com/metosin/malli/issues/125 . Btw I hope I'm not pestering too much - I simply try to make an educated attempt at improving perceived problems (which can last long - for example I use Schema at work and it kinda hurts to use an outdated tech that turned out to not be the best bet)#2021-03-1313:03lmergenwhat is the validator i can use for 'any collection, no matter the type' ? i don't care whether it's a sequence or vector or set, just that it's a collection.
what can i use for this? i wasn't able to find it#2021-03-1314:26ikitommi@lmergen maybe coll?#2021-03-1314:27lmergeni can use that like [:coll pos-int?] ?#2021-03-1314:30lmergentrying to figure out the best way to do this 🙂#2021-03-1314:31ikitommisadly, it's just the core predicate, so no child type checking. There is no :coll atm. Would be a oneliner to add#2021-03-1314:31ikitommiSee https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1858#2021-03-1314:32lmergenis there a specific reason why this is not in malli itself? as in, if i would send a PR to add this, would that be useful?#2021-03-1314:32ikitommino-one has asked, PR welcome (tests included)!#2021-03-1314:33lmergenof course 👍#2021-03-1314:33ikitommiit spans to generator, json schema, human errors etc. But, good examples :)#2021-03-1314:33lmergenthanks, i'll see what i can do#2021-03-1314:33lmergenyes it touches a lot of surface#2021-03-1314:44juhotepericoll? also matches maps, that has some effects on JSON-Schema implementation at least#2021-03-1315:19ikitommigood point. I guess there is not a predicate for non-map collection.#2021-03-1315:24nilernI imagine maps also having keys can lead to all sorts of confusion when implementing [:coll pos-int?]#2021-03-1315:24nilernWhat about :sequential?#2021-03-1315:25nilernOh that does not take sets#2021-03-1315:31nilernCan always do [:or [:sequential pos-int?] [:set pos-int?]]#2021-03-1315:33nilernThat does not support seqs though... and now I see nothing else does either.#2021-03-1315:37nilernMade some issues about that https://github.com/metosin/malli/issues/393 https://github.com/metosin/malli/issues/394#2021-03-1506:28fmnCurrently, is it possible to do this with parse ?
(s/def ::name-or-id (s/or :name string?
:id int?))
(s/conform ::name-or-id "abc")
;;=> [:name "abc"]
(s/conform ::name-or-id 100)
;;=> [:id 100]
#2021-03-1508:09ikitommi@funyako.funyao156 try [:orn [:name string?] [:id int?]]. There are named-branch variants for :or, :alt and :cat with +n in the name.#2021-03-1511:06fmn@ikitommi Thanks!#2021-03-1618:01raymcdermott@ikitommi trying to get the graphviz example working with dorothy but it's all a string. What do you use to render it?#2021-03-1618:12ikitommi@raymcdermott try println#2021-03-1618:54raymcdermottI meant to a PNG or whatever ...#2021-03-1619:39ikitommioh, that… 🙂 just dot from command line or the online version, e.g. https://sketchviz.com/#2021-03-1619:40ikitommilove the sketches#2021-03-1619:46ikitommibut, with dorothy, this seems to work:
(require '[dorothy.core :as d])
(->> (md/transform Order) (d/show!))
#2021-03-1620:26raymcdermottok ... I was holding it wrong#2021-03-1621:23raymcdermottalso m/walk ... thank you!!#2021-03-1622:11raymcdermotthaving said that#2021-03-1622:11raymcdermotthow do I use it to drop or add in Swagger attributes?#2021-03-1622:12raymcdermott(def Org-Ref
[:map {:title "Organisation name"}
[:ref {:swagger/description "Reference to the organisation"
:swagger/example "Acme floor polish, Houston TX"} Name]])#2021-03-1622:12raymcdermottI tried mu/update-properties but that only seems to operate at the top level#2021-03-1622:13raymcdermott[ cos they clutter up the graphics 🙂 ]#2021-03-1706:26ikitommi@raymcdermott you can walk + update-properties, but if you want to walk also the map-entries (not just map values), you should paramertise the walk to walk those too. same for :refs.#2021-03-1706:26ikitommisome tests on walking here: https://github.com/metosin/malli/blob/master/test/malli/util_test.cljc#L672-L795#2021-03-1706:27ikitommie.g. ::m/walk-refs & walk-schema-refs & ::m/walk-entry-vals.#2021-03-1706:28ikitommihopefully not too complex, balancing between “you can do anything with this” and “should be easy to do the simple stuff”#2021-03-1706:29ikitommialso, the dot-printer could have a option with schema->schema function to manipulate the schemas before printing?#2021-03-1709:04raymcdermottI will have to consider that once I get it all to work 🙂#2021-03-1717:17raymcdermottI have read the tests and am struggling to see how they help me to drop the swagger entries. I'm not even sure how to address them .... I know they are the the map in slot 1 of the key :ref but can see how to access that. None of them seem to show how to manipulate entries. And, I know it's my limitation but going to the schema processing itself is quite complex code to read out in my head.#2021-03-1717:20raymcdermottI was thinking that - since they are only things defined with maps, I could find a way to drop maps like I would do for some type with walk#2021-03-1717:39ikitommiI'll try it out#2021-03-1718:09ikitommioh, no easy way for that. will add something.#2021-03-1718:35raymcdermottthanks @U055NJ5CC#2021-03-1802:57yuhanHi, I'm just working through the Readme tutorial on the latest 0.3.0 and found that this example failed:
(m/validate
[:map
["status" [:enum "ok"]]
[1 any?]
[nil any?]
[::a string?]]
{"status" "ok"
1 'number
nil :yay
::a "properly awesome"})
; => true
Instead I get an exception:
1. Unhandled clojure.lang.ExceptionInfo
:malli.core/naked-keys-not-supported nil
{:type :malli.core/naked-keys-not-supported, :data nil}#2021-03-1803:00yuhanIs this a regression or API change? I couldn't find references to "naked keys" apart from internal impl code#2021-03-1805:29ikitommi@qythium it was a regression, fixed in master#2021-03-1808:19HankstenbergIs there a way to filter data by a schema? So if I have data of the form {:a 1 :b 2} and a schema of the form [:map [:a int?]] can I "apply" the schema to the data in order to get {:a 1}?
All I can think of right now is to use m/explain then parse the errors.#2021-03-1809:01ikitommi@raymcdermott would this be ok:
(def Org-Ref
[:map {:title "Organisation name"}
[:ref {:swagger/description "Reference to the organisation"
:swagger/example "Acme floor polish, Houston TX"} :string]
[:kikka [:string {:swagger {:title "kukka"}}]]])
(defn remove-swagger-keys [p]
(not-empty (apply dissoc p (into #{:swagger} (->> p (keys) (filter (comp #{:swagger} keyword namespace)))))))
(defn walk-properties [schema f]
(m/walk
schema
(fn [s _ c _]
(m/into-schema
(m/-parent s)
(f (m/-properties s))
(cond->> c (m/entries s) (map (fn [[k p s]] [k (f p) (first (m/children s))])))
(m/options s)))
{::m/walk-entry-vals true}))
(walk-properties Org-Ref remove-swagger-keys)
;[:map {:title "Organisation name"}
; [:ref :string]
; [:kikka :string]]#2021-03-1809:02ikitommie.g. walk the entrys, un-walk on the way back. apply f on all properties (entrys & schemas)#2021-03-1809:03ikitommi@roseneck you can transform the value using strip-extra-keys-transformer:
(m/decode [:map [:a int?]] {:a 1, :b 2} (mt/strip-extra-keys-transformer))
; => {:a 1}#2021-03-1809:09Hankstenberg@ikitommi perfect, thank you very much!#2021-03-1809:50raymcdermottyes @ikitommi that's very elegant#2021-03-1810:05robert-stuttafordis it possible to introspect the malli schema from inside an :error/fn fn ? i.e. i want to (first (m/children schema)) so i can print out the enum values in the error message#2021-03-1810:34ikitommi@robert-stuttaford sure, the fn takes the explain error map as argument, it has the :schema key.#2021-03-1810:35ikitommithere are lot of examples in malli.error#2021-03-1810:38ikitommihttps://github.com/metosin/malli/blob/master/src/malli/error.cljc#L75-L80#2021-03-1811:02robert-stuttafordwhen i try this, i get m/children isn't a thing, because it's SCI that's running this code#2021-03-1811:02robert-stuttaford(thank you for your quick response)#2021-03-1811:05robert-stuttafordExecution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:51).
Could not resolve symbol: m/children
#2021-03-1811:05robert-stuttafordis there a trick to get it to see malli?#2021-03-1811:08borkdude@robert-stuttaford just out of curiosity: what is your use case for malli + SCI?#2021-03-1811:08borkdudemalli could make this namespace available inside of sci#2021-03-1811:08ikitommithere is ::m/sci-options to override the bindings. The default bindings are:
(defn -default-sci-options []
{:preset :termination-safe
:bindings {'m/properties properties
'm/type type
'm/children children
'm/entries entries}})
have the sci-options changed? don’t seem to work anymre#2021-03-1811:09ikitommi:termination-safe is removed at least#2021-03-1811:09borkdudethe options have not been changed, but :bindings are only valid within the user namespace, always have been. it's better to use explicit :namespaces#2021-03-1811:09robert-stuttafordhonestly i'm using sci because malli is#2021-03-1811:09borkdudeyes :termination-safe has been removed for a while already, also documented in release notes#2021-03-1811:10robert-stuttafordall i'm doing is writing malli specs at the repl with :error/fn and i ran into an error 'sci not available', so i put it on the CP and onward i went#2021-03-1811:10ikitommi@robert-stuttaford if you don’t need the seriaization thing, just pass a real function.#2021-03-1811:10robert-stuttafordoh man. the fn is quoted. shit. sorry for the noise, fellas#2021-03-1811:10ikitommi🙂#2021-03-1811:10borkdudethat's what I thought :) maybe the error message should be : use sci for serialized schemas#2021-03-1811:10borkdudeor something#2021-03-1811:11ikitommiall properties which have functions as values use the malli.eval, which uses sci as default for quoted code.#2021-03-1811:12ikitommie.g.`:gen/fmap '(partial str "kikka_")`#2021-03-1811:12ikitommibut, what is the right way to bind those m/children into sci via options?#2021-03-1811:13borkdude{:namespaces {'malli.core {'children m/children}}}#2021-03-1811:15ikitommidoesn’t work either.#2021-03-1811:15ikitommi(defn -default-sci-options []
{:namespaces {'malli.core {'properties properties
'type type
'children children
'entries entries}}})#2021-03-1811:15borkdudedoesn't work = which error?#2021-03-1811:15ikitommiExecution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:50).
Could not resolve symbol: malli.core/chidren [at line 1, column 10]
#2021-03-1811:16ikitommi(defn evaluator [options fail!]
(let [eval-string* (dynaload/dynaload 'sci.core/eval-string* {:default nil})
init (dynaload/dynaload 'sci.core/init {:default nil})
fork (dynaload/dynaload 'sci.core/fork {:default nil})]
(fn [] (if (and @eval-string* @init @fork)
(let [ctx (init options)]
(fn eval [s] (eval-string* (fork ctx) (str s))))
fail!))))#2021-03-1811:16borkdude"malli.core/chidren" <- typo?#2021-03-1811:16ikitommi:face_palm:#2021-03-1811:16ikitommithanks, works like a charm#2021-03-1811:17ikitommiwhat about binding m -> malli.core for not breaking things?#2021-03-1811:29borkdudeyou can manually insert a (require '[malli.core :as m]) to ensure this works#2021-03-1811:30borkdudeor (alias 'm 'malli.core)#2021-03-1811:33ikitommilike:
(defn evaluator [options fail!]
(let [eval-string* (dynaload/dynaload 'sci.core/eval-string* {:default nil})
init (dynaload/dynaload 'sci.core/init {:default nil})
fork (dynaload/dynaload 'sci.core/fork {:default nil})]
(fn [] (if (and @eval-string* @init @fork)
(let [ctx (init options)]
(eval-string* ctx "(alias 'm 'malli.core)")
(fn eval [s] (eval-string* (fork ctx) (str s))))
fail!))))#2021-03-1811:33ikitommiseems to work#2021-03-1811:34ikitommi((m/eval 'm/type) :int)
; => :int
#2021-03-1811:34ikitommi🙇#2021-03-1810:52euccastrohow do I transform the keys of a schema? e.g., I have [:map [:a-k string?] [:b-k string?]] and I want to derive a schema that is the same but with snake-case keys: [:map [:a_k string?] [:b_k string?]]#2021-03-1811:21ikitommi@euccastro try m/walk with m/schema-walker check that the schema is a :map and recreate the children.#2021-03-1811:25euccastrothank you!#2021-03-1811:47robert-stuttafordis the best way to specify a literal value to use a single-item enum?#2021-03-1811:49delaguardo[:= value]#2021-03-1811:50robert-stuttafordnoice!#2021-03-1812:15borkdudeTried to use this to build a core.match like this ;)
https://gist.github.com/borkdude/26906ee15585ed5e1b7a8eda4cc1ee18#2021-03-2207:03robert-stuttafordnice man#2021-03-1816:35danielnealdumb question, how do you get a schema from the registry by its key#2021-03-1816:35danielnealI was thinking (malli/-schema malli/default-registry ::sq/sqid)#2021-03-1816:35danielnealbut -schema is private#2021-03-1816:37emccue[:map
[:field-a string?]
[:field-b [:one-of
[:map [:status [:= :not-asked]]]
[:map [:status [:= :loading]]]
[:map [:status [:= :failed]]]
[:map [:status [:= :success]
:value [:vector [:map [:id int?]]]]]]]
[:field-c [:one-of
[:map [:status [:= :not-asked]]]
[:map [:status [:= :loading]]]
[:map [:status [:= :failed]]]
[:map [:status [:= :success]
:value [:vector [:map [:id int?]]]]]]]
[:field-d [:set [:map [:id int?]]]]]#2021-03-1816:37emccuewhat would the idiomatic way to represent this be?#2021-03-1816:39emccuetrying on the playground it doesn't say its wrong#2021-03-1816:39emccue#2021-03-1816:39emccuebut it also doesn't produce any sample values#2021-03-1816:40ikitommi@emccue there is no one-of, you can use :or#2021-03-1816:41emccueoh.#2021-03-1816:41emccueyep that did it#2021-03-1816:42ikitommicould also be a :multi dispatching on :type .#2021-03-1816:44ikitommi@danieleneal try (m/deref (m/schema ::sq/said))#2021-03-1816:45emccueWhat is the benefit of multi schemas over explicit listing like that?#2021-03-1816:45ikitommi(m/deref ::sq/said) might work too.#2021-03-1816:47ikitommimight not be big difference, but performance. dispatch does one lookup to find the correct schema, :or does linear scan over all.#2021-03-1816:49emccueI think last question for now - what would be the best way to reuse a structure like this#2021-03-1816:49emccuejust a function that takes in the success value schema and returns the whole thing?#2021-03-1816:56danielneal@ikitommi, (m/deref ::sq/sqid) works thanks :thumbsup:#2021-03-1817:27danielnealis there a way of getting the schema walker to deref while walking? I've got a schema which is like [:map [:some-key :some-schema] [:another-key :another-schema]] where :some-schema and :another-schema are in the registry. I want to transform all the keys to snake case, but the schema walker doesn't descend into schema references.#2021-03-1817:35ikitommi@danieleneal see:
> e.g. ::m/walk-refs & ::m/walk-schema-refs & ::m/walk-entry-vals.#2021-03-1817:36ikitommiwalking respects those options, can't recall what does what. Please try, documentation PR welcome#2021-03-1817:36ikitommiI recall those are recursion safe#2021-03-1817:37ikitommie.g. stop of first deref of already walked reference#2021-03-1817:38danielnealoh cool#2021-03-1817:38danielnealthanks again!!!#2021-03-1821:21ikitommi@emccue one way to reuse is to use local registry:
[:map {:registry {:user/success [:map
[:status [:= :success]]
[:value [:vector [:map [:id int?]]]]]
:user/default [:map
[:status [:enum :not-asked :loading :failed]]]
:user/field [:multi {:dispatch :status}
[:success :user/success]
[:malli.core/default :user/default]]}}
[:field-a string?]
[:field-b :user/field]
[:field-c :user/field]
[:field-d [:set [:map [:id int?]]]]]#2021-03-1821:21ikitommihttps://malli.io/?value=%7B%3Afield-a%20%22kikka%22%2C%0A%20%3Afield-b%20%7B%3Astatus%20%3Aloading%7D%2C%0A%20%3Afield-c%20%7B%3Astatus%20%3Afailed%7D%2C%0A%20%3Afield-d%20%23%7B%7B%3Aid%209807%7D%7D%7D&schema=%5B%3Amap%20%7B%3Aregistry%20%7B%3Auser%2Fsuccess%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Astatus%20%5B%3A%3D%20%3Asuccess%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Avalue%20%5B%3Avector%20%5B%3Amap%20%5B%3Aid%20int%3F%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Fdefault%20%5B%3Amap%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Astatus%20%5B%3Aenum%20%3Anot-asked%20%3Aloading%20%3Afailed%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Ffield%20%5B%3Amulti%20%7B%3Adispatch%20%3Astatus%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Asuccess%20%3Auser%2Fsuccess%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amalli.core%2Fdefault%20%3Auser%2Fdefault%5D%5D%7D%7D%0A%20%5B%3Afield-a%20string%3F%5D%0A%20%5B%3Afield-b%20%3Auser%2Ffield%5D%0A%20%5B%3Afield-c%20%3Auser%2Ffield%5D%0A%20%5B%3Afield-d%20%5B%3Aset%20%5B%3Amap%20%5B%3Aid%20int%3F%5D%5D%5D%5D%5D#2021-03-1821:22ikitommicould be of course global registry too.#2021-03-1821:23emccueI mean like, this is analagous to a typed enum from other langs#2021-03-1821:23emccueso i would in my brain go#2021-03-1821:23emccue(remote-data [:map [:id int?]])#2021-03-1821:23emccueand reuse the pattern#2021-03-1821:24ikitommioh, sure. it’s just data, so a function liike that is the way to do it.#2021-03-1821:29emccuebut by the same token, if i have a spec like this in a namespace#2021-03-1821:29emccue(def user [:map [:name string?]])
#2021-03-1821:29emccueI shouldn't use it like this#2021-03-1821:30emccue(def school [:vector other.ns/user])
#2021-03-1821:30emccueright?#2021-03-1821:30emccuebecause then it will get "flattened" and the errors won't be as good#2021-03-1821:32nilernThat's how we used to do it with Plumatic Schema#2021-03-1821:35nilernWe wanted to enable that style too, you may give up some serialization benefits but get to use normal defs etc.#2021-03-1821:40nilernThe registry names don't play any role in e.g. explainer actually#2021-03-1821:49emccueso then a local registry is basically equivalent to a let?#2021-03-1821:49emccue(but "runs" in the schema?)#2021-03-1903:12willierspec->malli was mentioned here a while ago. is anyone aware if such a thing exists?#2021-03-1906:03ikitommi@emccue currently it’s a let. Could be something else too. What kind of better errors would you expect? The reference information is available over there, just not used I guess in explain:
(m/explain
[:map {:registry {"ID" :int}}
[:id "ID"]]
{:id "123"})
;{:schema [:map {:registry {"ID" :int}} [:id "ID"]],
; :value {:id "123"},
; :errors (#Error{:path [:id], :in [:id], :schema :int, :value "123"})}#2021-03-1906:03ikitommi@willier not yet. interested in doing? 😉#2021-03-1908:43willierhmm, probably need a black belt in macro-fu to do this job#2021-03-1908:44nilernOr a spec-tools belt?#2021-03-1908:56borkdude@willier you can inspect specs at runtime. spec-tools has something called a walker and coax is a library from exoscale to do coercion based on specs, I think they are doing something similar#2021-03-1908:58ikitommispec-tools has both walker and a visitor 🙂 visitor is the right tool for the job i believe: https://cljdoc.org/d/metosin/spec-tools/0.10.5/doc/spec-visitor#2021-03-1908:58ikitommijson schma with it: https://cljdoc.org/d/metosin/spec-tools/0.10.5/doc/json-schemas#2021-03-1909:00willieroh interesting... i will have a look into it - thanks!#2021-03-1909:01ikitommialso, there are progression tests in spec-tools for broken forms: if the core would be fixed: https://github.com/metosin/spec-tools/blob/master/test/cljc/spec_tools/visitor_all_test.cljc#2021-03-1909:03ikitommioh, no more, tests disabled and the last form is fixed. great!#2021-03-1909:43danielnealIf you have a map that references other schemas, like this [:map ::some/ref ::some-other/ref] how do you make ::some/ref and ::some-other/ref optional?#2021-03-1909:47danielnealI tried [:and {:optional true} ::some/ref] but that was invalid#2021-03-1909:49nilernYou just have to use the sugar-free [:map [::some/ref {:optional true} ::some/ref]]
You could get clever and [:map (#(vector % {:optional true} %) ::some/ref)] or name and reuse that little fn#2021-03-1909:52nilern[:map ::some/ref] is just some sugar in :map, we don't have "first-class properties" beyond that. And :optional is specific to map entries while :maybe does something else.#2021-03-1909:53danielnealaha ok thanks 🙂#2021-03-1910:05ikitommithis should also work: [:map [::some/ref {:optional true}]], no need to repeat the schema for refs.#2021-03-1910:21danielnealnice!#2021-03-1910:59danielnealIf I'm walking a schema and want to convert all schemas of type ::sq/ref into :string, but I also want to deref all other refs, how do I do this/
This is what I've got so far:
(defn output-schema
[schema]
(malli/walk
schema
(malli/schema-walker
(fn [schema]
(malli/type schema)
(cond
(= schema ::sq/ref) :string
(= (malli/type schema) :map)
.... do some other stuff
:else schema)))
{::malli/walk-schema-refs true
::malli/walk-refs true}))
But the schema is expanded by the time the check happens, so it fails#2021-03-1911:00ikitommiwhat does (malli/walk schema (malli/schema-walker identity) {::malli/walk-schema-refs true, ::malli/walk-refs true}) do?#2021-03-1911:11danielnealyep, that expands all#2021-03-1911:01ikitommithat might expand all automatically#2021-03-1911:09ikitommi(m/walk
[:schema
{:registry {"Country" [:map
[:name [:enum :FI :PO]]
[:neighbors [:vector [:ref "Country"]]]]
"Burger" [:map
[:name string?]
[:description {:optional true} string?]
[:origin [:maybe "Country"]]
[:price pos-int?]]
"OrderLine" [:map
[:burger "Burger"]
[:amount int?]]
"Order" [:map
[:lines [:vector "OrderLine"]]
[:delivery [:map
[:delivered boolean?]
[:address [:map
[:street string?]
[:zip int?]
[:country "Country"]]]]]]}}
"Order"]
(m/schema-walker #(mu/update-properties % assoc :type (m/type %)))
{::m/walk-schema-refs true})
;[:schema
; {:registry {"Country" [:map [:name [:enum :FI :PO]] [:neighbors [:vector [:ref "Country"]]]],
; "Burger" [:map
; [:name string?]
; [:description {:optional true} string?]
; [:origin [:maybe "Country"]]
; [:price pos-int?]],
; "OrderLine" [:map [:burger "Burger"] [:amount int?]],
; "Order" [:map
; [:lines [:vector "OrderLine"]]
; [:delivery
; [:map [:delivered boolean?] [:address [:map [:street string?] [:zip int?] [:country "Country"]]]]]]},
; :type :schema}
; [:malli.core/schema
; {:type :malli.core/schema}
; [:map
; {:type :map}
; [:lines
; [:vector
; {:type :vector}
; [:malli.core/schema
; {:type :malli.core/schema}
; [:map
; {:type :map}
; [:burger
; [:malli.core/schema
; {:type :malli.core/schema}
; [:map
; {:type :map}
; [:name [string? {:type string?}]]
; [:description {:optional true} [string? {:type string?}]]
; [:origin
; [:maybe
; {:type :maybe}
; [:malli.core/schema
; {:type :malli.core/schema}
; [:map
; {:type :map}
; [:name [:enum {:type :enum} :FI :PO]]
; [:neighbors [:vector {:type :vector} [:ref {:type :ref} "Country"]]]]]]]
; [:price [pos-int? {:type pos-int?}]]]]]
; [:amount [int? {:type int?}]]]]]]
; [:delivery
; [:map
; {:type :map}
; [:delivered [boolean? {:type boolean?}]]
; [:address
; [:map
; {:type :map}
; [:street [string? {:type string?}]]
; [:zip [int? {:type int?}]]
; [:country
; [:malli.core/schema
; {:type :malli.core/schema}
; [:map
; {:type :map}
; [:name [:enum {:type :enum} :FI :PO]]
; [:neighbors [:vector {:type :vector} [:ref {:type :ref} "Country"]]]]]]]]]]]]]#2021-03-1911:10ikitomminot sure if that’s near what you want to do.#2021-03-1911:13danielnealAh I think I said :type wrong, the type is :malli.core/schema, what I'm checking for is the schema itself to be ::sq/ref#2021-03-1911:14danielnealSo is what you're suggesting to do two walks, one to capture information and put it in the properties, and then a second to do the other transformations?#2021-03-1911:18danielnealhmm#2021-03-1911:20danielnealI suppose I could put some property on the ::sq/ref schema itself, but it feels like I'm missing a trick#2021-03-1911:28nilernTo do it in one traversal you would need a more Visitor-like prewalk where you manually deref the ones you want.
I think that can be done with -walk and Walker but only schema-walker and walk are documented and stable ATM#2021-03-1911:31danielnealyeah that makes sense, you kind of prewalk it and choose whether to deref or swap as you descend#2021-03-1912:38ikitommiok, I guess I missed the original point. But there are public walkers like https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L37-L51#2021-03-1915:00danielnealis it safe to use those functions and protocols?#2021-03-1915:03nilernhttps://github.com/metosin/malli/#alpha#2021-03-1915:03nilern> extender api: public vars, name starts with -, e.g. malli.core/-collection-schema. Not needed with basic use cases, might evolve during the alpha, follow https://github.com/metosin/malli/blob/master/CHANGELOG.md for details#2021-03-1915:27danielnealnice, i see#2021-03-1911:31danielnealyeah that makes sense, you kind of prewalk it and choose whether to deref or swap as you descend#2021-03-1911:32danielnealthanks#2021-03-1917:36raymcdermottdoing the graphviz thing is a bit whack it turns out cos it passes through all of the constraints to be visualised and it soon gets ugly#2021-03-1917:36raymcdermott#2021-03-1917:38raymcdermottSo I guess you really have to pare back everything to the simplest of all possible models for nice visuals 🙂#2021-03-1917:42nilernMaybe we could have custom visualization props like we have :error/message?#2021-03-1918:45raymcdermottyes, or like {:swagger}#2021-03-1918:48raymcdermottif one had a Member, it would be nice for example to have {:viz/parent Org} on the map and {:viz/type string} on the name property#2021-03-1918:49raymcdermottand then one could select for those properties when transforming to DOT#2021-03-1921:54raymcdermott@ikitommi I might have been going about this wrong, so maybe this is now a better explanation of what I would like to have ...#2021-03-1921:55raymcdermott(def Id [:re #"^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$"])
(def Name [:string {:min 3 :max 255}])
(def org-ref :ref)
(def org-ref-viz [:map [org-ref :string]])
(def org-ref-description "Reference (name or ID) of the organisation")
(def org-ref-valid [:map {:title org-ref-description} [org-ref Name]])
(def org-ref-swagger {:swagger {:description org-ref-description
:example "Acme tech, Houston TX"}})
(def Org-Ref [:map {:title (name org-ref)}
[org-ref org-ref-swagger Name]])
(def org-id :id)
(def org-id-viz [:map [org-id :string]])
(def org-id-description "The organisation ID")
(def org-id-valid [:map [org-id Id]])
(def org-id-swagger {:swagger {:description org-id-description
:example (->id)}})
(def Org-Id [:map {:title (name org-id)}
[org-id org-id-swagger Id]])
(def org-viz {"Org" (mu/merge org-id-viz org-ref-viz)})
(def Org (mu/merge Org-Id Org-Ref))#2021-03-1921:56raymcdermottI can't find a way to create org-ref-full by combining org-ref-valid with org-ref-swagger#2021-03-1921:57raymcdermottI have added org-ref-viz as that's what I would like to derive from org-ref-valid#2021-03-1921:59raymcdermottAt the moment I am OK with doing it the way as it is above cos there is still a lot of value but obviously there seems to be quite a bit of boiler-plate#2021-03-2111:02Yevgeni TsodikovHey,
This might be a silly question.
Is it possible to validate a map while ignoring optional fields if they are invalid?
for example it would return false here:
(m/validate [:map
[:a string?]
[:b {:optional true} int?]]
{:a "Hey" :b "Nope"})
=> false
But there would be a way to conform a map with invalid optional fields and “sanitize” the map
; This made-up function would return {:a "Hey"}
(m/conform [:map
[:a string?]
[:b {:optional true} int?]]
{:a "Hey" :b "Nope"})#2021-03-2111:15ikitommiDo you mean the first should return true?#2021-03-2112:00HankstenbergHm, shouldn't a validation with an open map that just containts the schema for :a do the trick? Or let b be any? Under what circumstances should b be an int?#2021-03-2112:45ikitommi> Is it possible to validate a map while ignoring optional fields if they are invalid?
for that, yes, you can transform the schema before validation, just convert optional values to :any. but if you would like to strip away invalid values, you could:
1. run m/explain on data
2. recursively remove all values in :errors :in path. the invalid paths are part of the explain result:
(m/explain
[:map
[:a string?]
[:b {:optional true} int?]]
{:a "Hey" :b "Nope"})
;{:schema [:map [:a string?] [:b {:optional true} int?]],
; :value {:a "Hey", :b "Nope"},
; :errors (#Error{:path [:b], :in [:b], :schema int?, :value "Nope"})}#2021-03-2113:11Hankstenberg@U055NJ5CC That's something I'd like to be able to do too. Is there an idiomatic way to do that?#2021-03-2113:15Yevgeni Tsodikov> Do you mean the first should return `true`?
Yes 🙂
> recursively remove all values in `:errors` `:in` path. the invalid paths are part of the explain result:
The :optional data is missing from the :schema field, how can I know which field is safe to remove and which field makes the map actually invalid?#2021-03-2113:19Yevgeni TsodikovMy scenario is an API with some optional fields.
If the client doesn’t send then -> no worries.
If the client sent some invalid optional fields, in some cases I’d like to pass the request and not fail it.
(Such cases may be that the client is a mobile device with a crappy sdk, which might report invalid data for optional fields)#2021-03-2113:34Yevgeni TsodikovAlso -
> Do you mean the first should return `true`?
Not without some hints or options.
; This is correct
(m/validate [:map
[:a string?]
[:b {:optional true} int?]]
{:a "Hey" :b "Nope"})
=> false
; Additional options
(m/validate [:map
[:a string?]
[:b {:optional true} int?]]
{:a "Hey" :b "Nope"}
{:fail-on-optional? false)
=> true#2021-03-2112:45ikitommi> Is it possible to validate a map while ignoring optional fields if they are invalid?
for that, yes, you can transform the schema before validation, just convert optional values to :any. but if you would like to strip away invalid values, you could:
1. run m/explain on data
2. recursively remove all values in :errors :in path. the invalid paths are part of the explain result:
(m/explain
[:map
[:a string?]
[:b {:optional true} int?]]
{:a "Hey" :b "Nope"})
;{:schema [:map [:a string?] [:b {:optional true} int?]],
; :value {:a "Hey", :b "Nope"},
; :errors (#Error{:path [:b], :in [:b], :schema int?, :value "Nope"})}#2021-03-2114:06ikitommi#2021-03-2114:07ikitommiping @U01PU4APSKV#2021-03-2114:22Yevgeni TsodikovThat’s great, thanks @U055NJ5CC!#2021-03-2114:35Yevgeni TsodikovIs there a way to conform/remove the invalid optional fields?#2021-03-2114:51ikitommiyes. Many ways to do that :
1) attach a custom transformer to the new :any fields to strip those away
2) in case of error, call m/explain and remove all values from paths that have error#2021-03-2114:51ikitommidon't have time to write an example, but definitely doable 😉#2021-03-2116:18Yevgeni TsodikovWhat’s your opinion on something like:
(defn allow-invalid-optional-values [schema]
(m/walk
schema
(m/schema-walker
(fn [s]
(cond-> s
(m/entries s)
(mu/transform-entries
(partial map (fn [[k {:keys [optional] :as p} s]]
(if optional
[k
(assoc p :original-spec s
:decode/remove-invalid-fields {:compile (fn [schema _]
(fn [x]
(if-let [original-spec (:original-spec (m/properties schema))]
(when (m/validate original-spec x)
x)
x)))})
:any]
[k p s])))
(m/options s)))))))
(-> [:map
[:a string?]
[:b {:optional true} int?]]
allow-invalid-optional-values
(m/decode
{:a "Hey" :b "Nope"}
(mt/transformer {:name :remove-invalid-fields})))
=> {:a "Hey", :b nil}
I don’t like having multiple transformers, though.
How can I unify them? Similarly to what malli offers with mt/strip-extra-keys-transformer ?#2021-03-2114:12ikitommias the slack history rolls out fast, pasted the example into https://github.com/metosin/malli/blob/master/docs/tips.md#allowing-invalid-values-on-optional-keys#2021-03-2115:37Yevgeni TsodikovThe allow-invalid-optional-values function is missing the (m/options s) arg of mu/transform-entries#2021-03-2116:31ikitommihttps://github.com/metosin/malli/commit/da6366dd845b300923a5aec28eae96f15a8c9f71#2021-03-2116:15respatializedhi! I've been really impressed with malli so far. really appreciate all the work you've put into it.
I'm trying to generate sample values from a schema, and I'm encountering a case where :orn fails and :altn succeeds:
(mg/generate [:altn [:bool boolean?] [:num int?]] {:seed 20})
=> [true]
(mg/generate [:orn [:bool boolean?] [:num int?]] {:seed 20})
=> Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:16).
:malli.generator/no-generator {:schema [:orn [:bool boolean?] [:num int?]], :options {:seed 20}}
Is this expected at this stage? Is there additional implementation for :orn generators that still needs to be done, or is this a bug?#2021-03-2116:17respatializedsimple :or also succeeds on this case:
(mg/generate [:or boolean? int?] {:seed 20})
=> true
#2021-03-2116:49ikitommi@afoltzm 🙇 , see https://github.com/metosin/malli/pull/400#2021-03-2116:54ikitommimerged in master#2021-03-2117:04respatializedwow, that was fast! thanks!#2021-03-2117:05borkdudeit's weekend, the time where magic OSS happens#2021-03-2118:09ikitommipushed out [metosin/malli "0.3.1"], finally with a working cljdoc - https://cljdoc.org/d/metosin/malli/0.3.1/doc/changelog.#2021-03-2207:00robert-stuttafordhaha just my luck, a week after i get started, the docs improve 😆 it looks great!#2021-03-2207:01robert-stuttafordbtw @ikitommi i ended up chunking the results of sets of 100 item m/provide into an atom and then m/union ing them together afterward. i ended up with a fair bit of [:or [:or [:or ....]]] but that was easy to remedy with a postwalk 🙂#2021-03-2207:33ikitommi@robert-stuttaford good to hear. the right solution is to add few new Protocols (e.g. Inferrer and Provider) and attach the latter to the reified IntoSchemas: each instance know how to construct itself given a value and options. Internally using type/class-based lookup tables, it should be 2-4 orders of magnitude faster, and pluggable. Have an pre-initial draft somewhere 🙂#2021-03-2207:57robert-stuttafordthat sounds sane!#2021-03-2210:06Panel(malli/encode
[:or [:map
[:password string?]
[:password2 string?]]
[:fn {:error/message "passwords don't match"}
'(fn [{:keys [password password2]}]
(= password password2))]]
{:password "ok"}
(mt/key-transformer {:encode name}))
=> {:password "ok"}
(malli/encode
[:and [:map
[:password string?]
[:password2 string?]]
[:fn {:error/message "passwords don't match"}
'(fn [{:keys [password password2]}]
(= password password2))]]
{:password "ok"}
(mt/key-transformer {:encode name}))
=> {"password" "ok"}
Why doesn't encode work on the first one but does on the second, related to using :or instead of :and I guess.#2021-03-2210:11ikitommiI believe neither of the examples doesn’t do anything, they are just just no-op.#2021-03-2210:12ikitommioh, wait.#2021-03-2210:14nilernencode does not validate and those inputs are invalid, so nasal demons#2021-03-2210:15ikitommiactually:
(-transformer [this transformer method options]
(let [this-transformer (-value-transformer transformer this method options)]
(if (seq children)
(let [transformers (mapv #(or (-transformer % transformer method options) identity) children)
validators (mapv -validator children)]
(-intercepting this-transformer
(if (= :decode method)
(fn [x]
(reduce-kv
(fn [x i transformer]
(let [x* (transformer x)]
(if ((nth validators i) x*) (reduced x*) x)))
x transformers))
(fn [x]
(reduce-kv
(fn [x i validator] (if (validator x) (reduced ((nth transformers i) x)) x))
x validators)))))
(-intercepting this-transformer))))#2021-03-2210:15nilernAlthough in terms of the implementation I think both of those should be no-ops#2021-03-2210:15ikitommiit does validate and the first one fails -> doesn’t transform#2021-03-2210:18nilernAh yes so :or and seqex schemas validate when encoding because they have to pick a branch but nothing else (e.g. :and does)
But that is implementation detail#2021-03-2210:23nilernThe recommended flow is first validate, then encode if ok, else explain
Ceterum censeo, maybe some day we will have a thing that does all of that in one pass like Plumatic coerce#2021-03-2210:25PanelThanks for the explanations !#2021-03-2210:45nilernThis is my pet peeve and now I finally added an issue https://github.com/metosin/malli/issues/404
But it is a big decision and feature#2021-03-2214:18robert-stuttafordis it possible for malli to report all errors instead of stopping on the first one?#2021-03-2214:19nilernexplain(er) should report all the errors. What are you seeing / not seeing?#2021-03-2214:20robert-stuttafordah, apologies. i'm using explain -> with-spell-checking -> humanize#2021-03-2214:21robert-stuttafordseems humanize is dropping it#2021-03-2214:21ikitommireally? should not.#2021-03-2214:23robert-stuttafordlemme make reaaaal sure first, don't want to waste your time 😅#2021-03-2214:29robert-stuttafordok, it's not humanize. in the explain, the last error that i get is a :malli.core/input-remaining . when i fix that issue, then i get a new previously unreported error further down (happens to be the same issue: a kv-pair that should be one map level up)#2021-03-2214:30robert-stuttafordso, is there some way have the whole sequence reported on at once?#2021-03-2214:31robert-stuttafordbasically, i'd like to use malli specs to provide a structural EDN linter in a web-based cms code editor. but for this one thing, it's working, beautifully#2021-03-2214:33robert-stuttafordi am getting other errors from other collections, but the same issue manifests, in that i only get one error per such collection.
i guess the question i'm asking is, how do i make :sequential and/or :+ check all elements#2021-03-2214:34ikitommiOh, the partial seqexp, no reporting / transformation on partially matched sequences atm. https://github.com/metosin/malli/issues/387#2021-03-2214:34robert-stuttafordok so this is because i'm using :+ rather than :sequential?#2021-03-2214:35ikitomminot sure, could you give an example with actual and expected result?#2021-03-2214:36robert-stuttafordok, i switched to sequential, and this is the reason - thank you!#2021-03-2214:36nilernYou should use e.g. :sequential instead of :*/`:+` if you can#2021-03-2214:36robert-stuttafordi can live without ensuring the sequence has at least one item#2021-03-2214:37nilern[:sequential {:min 1} :int]#2021-03-2214:40robert-stuttafordok rad i've got what i need. malli is lovely#2021-03-2214:41robert-stuttafordnow to use edamame's edn metadata stuff to wire up go-to-line links#2021-03-2214:42nilernIn general the seqex engine can't "resynchronize" after an erroneous element.
We could have some optimizer that recognizes [:+ :int] and transforms it to [:sequential {:min 1} :int] but it would add complexity and unpredictability...#2021-03-2214:42robert-stuttafordtotally fair#2021-03-2214:45ikitommidid a malli + edamame poc some time ago, can’t recall was it any good, but pushed just as a gist: https://gist.github.com/ikitommi/0e5c4e48d8aeb7dd176128856ecdacb5#2021-03-2214:45ikitommisomething @borkdude was asking I think.#2021-03-2214:47borkdudewhat I tried to do there is "complect" error message with locations directly#2021-03-2214:48borkdudebut I think you can just do explain and then look up the vals in the paths which have locations (or maybe they are already in the errors) and then postwalk the result to add the locations to the error messages#2021-03-2214:50robert-stuttafordthat second thing is what i'm doing#2021-03-2214:56robert-stuttafordyeah @ikitommi your gist is the beautiful general version of the ugly hack i'm busy writing. going to start again with yours, thank you#2021-03-2214:57nilernThat code looks familiar :thinking_face:#2021-03-2214:58borkdude@robert-stuttaford I think the second thing is probably the easiest?#2021-03-2215:02ikitommilooking at the code, it has transduce, I never use that. I can’t recall the origins of the code, just from one of the zillion scratch files. could be @nilern s work too :thinking_face:#2021-03-2215:06nilernLooking at the Metosin Slack history I did the unwrap and collect bits and remarked that the prewalk makes it O(n^2) but it only matters if there are large map keys or set members which hardly ever happens...#2021-03-2215:08nilernAnyway it took a while to write so it would be cool if it found some use#2021-03-2306:21robert-stuttafordthanks all, i'll report back on how it goes!#2021-03-2308:41ikitommi@raymcdermott I can look your thing today/tomorrow.#2021-03-2311:33ikitommi@d.ian.b no inbuilt mapping for fn?. But, you can:
(m/validate [:fn fn?] (partial +))
; => true
#2021-03-2311:34Ian Fernandezyeah#2021-03-2311:34ikitommialso:
(m/explain
[:=> [:cat [:* [:int {:gen/min -1000, :gen/max 1000}]]] :int]
(partial +)
{::m/function-checker mg/function-checker})
; => nil#2021-03-2311:34Ian FernandezI've made it right now haha#2021-03-2311:34ikitommi(m/explain
[:=> [:cat [:* [:int {:gen/min -1000, :gen/max 1000}]]] [:int {:min 1}]]
(partial +)
{::m/function-checker mg/function-checker})
;{:schema [:=> [:cat [:* [:int #:gen{:min -1000, :max 1000}]]] [:int {:min 1}]],
; :value #object[clojure.core$_PLUS_ 0x6edca2b1 "#2021-03-2311:35Ian Fernandeznice!#2021-03-2311:57Ian FernandezHow I can use a register of the (m/function-schemas) registry, onto m/validate?#2021-03-2316:06Ian Fernandez(get-in (m/function-schemas) [(quote *ns*) 'foo :schema])#2021-03-2316:29ikitommi@d.ian.b good question, atm, no easy way. Idea with the function registry is that there will be a instrument kinda thing, that will wrap a) some b) all registered function schmaas like Orchestra - running input & output validation. Could be also used to emit generated data based only on the function definitions. Ideas and PRs welcome on that.#2021-03-2316:30ikitommialso, did a spike on infferring schemas from normal vars. you get useful guesses pretty easily, better with tools.analyzer & clj-kondo and I guess. really good with core.typed.#2021-03-2316:36ikitommithe new kw-varargs thing would work nicely with global registry, my guess is that the core team will plug into that with spec. e.g. given a function:
(defn doit [& {:domain.user/keys [id name]}] [id name])
… running (m/collect #'doit) would infer a [:map :domain.user/id :domain.user/name] out of it, register it as a malli function schma, after (m/instrument my-registry) saying:
(doit :domain.user/id 1, :domain.user/name "kikka")
would cause it to run validation.#2021-03-2316:38ikitommigiven there is few hours extra time, I would write a sample code so that I could prove a point that it’s doable and awesome.#2021-03-2316:45raymcdermottI don't seem to be able to merge maps with [:and] constraints#2021-03-2316:47raymcdermott(def x [:and [:map
[:start int?]
[:end int?]]
[:fn (fn [{:keys [start end]}]
(< start end))]])
=> #'user/x
(def y [:map
[:here int?]
[:there int?]])
=> #'user/y
(require '[malli.util :as mu])
=> nil
(mu/merge y x)
=> [:and [:map [:start int?] [:end int?]] [:fn #object[user$fn__4910 0x40fa91ef "#2021-03-2316:47raymcdermottis this expected?#2021-03-2316:54borkdudeand should merge combine the predicate in another predicate which ands those if both maps have predicates?#2021-03-2316:55raymcdermottit should invoke both functions ... maybe the order would not be predictable but I'll take that#2021-03-2316:55borkdudeprobably the order of merge args#2021-03-2316:56borkdudethis could lead to funny problems, like what if you merge in a closed map in an open map, probably the resulting map should be open?#2021-03-2316:59borkdudeor should merge consider predicates and other properties like metadata, which is ignored in merge args?#2021-03-2317:00borkdude(with-properties-of (merge x y) y)#2021-03-2317:54raymcdermottmerge takes the last as the 'winner' so I think that would be the most idiomatic#2021-03-2317:54raymcdermottbut I agree that merging things that are not maps is tricky#2021-03-2319:19ikitommiPlumatic dropped s/both in favour of s/constrained just because the first is not a good idea: “apple and fruit and a car” please. Currently :and already kinda means “the first thing constrained with the rest” as we pick the generator from first and then constraint with the rest using gen/such-that. Given that, we could make :and mergable, would merge with the first and keep the rest as extra leaves of :and? e.g.
[:map ::x]
[:map ::y]
; => [:map ::x ::y]
[:map ::x]
[:and [:map ::y] map?]
; => [:and [:map ::x ::y] map?]
[:and [:map ::x] map?]
[:map ::y]
; => [:and [:map ::x ::y] map?]
[:and [:map ::x] map?]
[:and [:map ::y] map?]
; => [:and [:map ::x ::y] map? map?]
[:and [:map ::x]]
map?
; => map?
would that be … more correct?#2021-03-2319:21ikitommihaving [:and [:map …] [:fn …]] is quite common, having it non-mergable is a bummer.#2021-03-2421:19raymcdermottif both maps have [:and [:map ..][fn...] it could be rejected. If one has an [:and ...] and the other doesn't can't you still merge the maps? I am probably under thinking it 🙂#2021-03-2322:07PanelAnyway to have a dynamic default ? Like if I want a default timestamp to be generated when runing decode with default-value-transformer ?#2021-03-2323:17Panel(malli/decode
[:map {:registry
{:inst (m/-simple-schema
{:type :inst
:pred inst?})}}
[:time :inst]
[:id1 :uuid]
[:id2 :uuid]]
{}
(mt/default-value-transformer
{:defaults {:inst (constantly (rand-int 100))
:uuid (constantly (char (rand-int 100)))}}))
Found something that works for me, but the default are generated once per type so for example if you need two different uuid it won't work.#2021-03-2323:34Panel(mu/optional-keys
[:map {:registry
{:inst (m/-simple-schema
{:type :inst
:pred inst?})}}
[:time :inst]
[:id :uuid]])
This throw a java.lang.StackOverflowError#2021-03-2401:56PanelIt does work if the registry is passed in the options map
(mu/optional-keys
[:map
[:time :inst]
[:id :uuid]]
{:registry
(mr/composite-registry
m/default-registry
{:inst (m/-simple-schema
{:type :inst
:pred inst?})})})#2021-03-2413:16respatializedIs there a preferred way to ensure that a seqex schema is a given collection type? right now I can use [:and vector? [:catn [:bool :boolean] [:int :int]]], but I was wondering if there was a more canonical or performant way of doing it (generators often fail to satisfy the predicate, for example).#2021-03-2413:23nilernNo. I think it only matters for generators and :and is in general not generator-friendly (currently the generator is a gen/such-that, which is pretty bad but how much better can we even get?)#2021-03-2413:25nilern(For that particular example I would probably use [:tuple :boolean :int] but you might really have something that actually needs seqex)#2021-03-2413:29respatializedyeah, that was a deliberately simplified example (the actual motivating use case is validation on hiccup forms). I think I'll need to take a crack at custom generators for this use case.#2021-03-2413:33nilernI think I've seen some discussion of this previously and it should be simple to add. So maybe file an issue as well?#2021-03-2413:51nilernIt is unfortunate that we have to add non-orthogonal props like [:string {:min 5}] and now maybe [:catn {:type :vector}] to get decent generators#2021-03-2414:04respatializedminimallist has an additional https://github.com/green-coder/minimallist/blob/f10ebbd3c2b93e7579295618a7ed1e870c489bc4/src/minimallist/helper.cljc#L72 that does this, minimallist.helper/in-vector, but it chooses a map-based representation for schemas that allows additional metadata of :coll-type to be assoc'd to the schema. Unclear how something similar would work in malli.#2021-03-2414:08nilernIt would just add that to props but the seqex schemas still have to be updated to use that information#2021-03-2518:38ikitommifirst kinda BREAKING change in the public api. m/parse on :multi now returns the branch information. It’s a fix, but still:
(def Multi
[:multi {:dispatch :type}
[:user [:map [:size :int]]]
[::m/default :any]])
(m/parse Multi {:type :user, :size 1})
; => [:user {:type :user, :size 1}]
(m/parse Multi {:type "sized", :size 1})
; => [:malli.core/default {:type "sized", :size 1}]
(->> {:type :user, :size 1}
(m/parse Multi)
(m/unparse Multi))
; => {:type :user, :size 1}#2021-03-2522:21ikitommi+5 loc later (including comments): :and merges on first child! comments welcome on the results - https://github.com/metosin/malli/pull/405#2021-03-2608:11ikitommiping @U04V5V0V4#2021-03-2616:20raymcdermottI thought it read 5k loc ... so 😅#2021-03-2616:21raymcdermottI have added a quick comment. Thanks for the feature#2021-03-2907:14Yevgeni TsodikovAwesome MR! Thanks 🙏#2021-03-2910:34ikitommimerged in master#2021-03-2621:15borkdudeI'm in the process of designing a task-executor DSL.
Example:
{:tasks {:clean [:shell "rm" "-rf" "target"]}}
I want to be able to give a task a description and possibly more options. So I thought I can do:
{:tasks {:clean [:shell {:task/description "foo"} "rm" "-rf" "target"]}}
but this reads a bit weird, because I expect :shell to get shell-related options and not some general task options.
One alternative I came up with:
{:tasks {:clean ^{:task/description "foo"} [:shell "rm" "-rf" "target"]}}
A bit ugly maybe, but semantically correct: the metadata is about the task. However, processing EDN with metadata, it can be done, but it might lead to confusion.
The other alternative, an explicit :task wrapper if you want to provide options:
{:tasks {:clean [:task {:description "foo"} [:shell "rm" "-rf" "target"]]}}
I wonder if you came across similar problems when designing malli, I'd love to hear your feedback.#2021-03-2621:26borkdudeI think malli more or less ended up with:
{:tasks {:clean [:shell {:description "Clean all the things"} "rm" "-rf" "target"]}}
right?#2021-03-2710:29borkdudeI think I'll land on:
{:tasks {:clean [:shell {:description "foo"} "rm -rf target"]}}#2021-03-2710:29borkdudethanks duckie :)#2021-03-2711:36PanelJust curious if you are using Malli to make the DSL ?#2021-03-2711:37borkdudenot really no#2021-03-2711:42borkdudealthough malli would probably be a good fit for validation#2021-03-2713:34borkdudeLanded on this one:
{:tasks {:clean {:description "Foo" :task [:shell "rm -rf target"]}}}
#2021-03-2713:35borkdudeNot hiccup as usual, but I think the cleanest so far#2021-03-2714:17ikitommi@borkdude looks good, I would have suggested a map too, keeps the concepts separated.#2021-03-2714:18ikitommiMalli also has the generic map-syntax and have explored a third, simpler (clj-fx-like) map-syntax.#2021-03-2714:23ikitommiNeed to pick a "better" map-syntax for malli 1.0.0, but will be first-class support, in malli.core. One can't mix syntaxes (terse/hiccup and map) freely, just one allowed per AST fragment.#2021-03-2714:56borkdude@ikitommi what is clj-fx-like map syntax? would love to see it#2021-03-2904:58ikitommi@borkdude wrote an issue of the syntax, with examples: https://github.com/metosin/malli/issues/406#2021-03-3007:22Adam HelinsIs there a way of generating sorted maps using :map-of ? Spec allows that and it is indeed very useful when working with sorted colls.#2021-03-3014:32nilernI don't think there is any support for sorted colls specifically#2021-03-3019:24Adam HelinsInteresting, i can't find anything, no issue, nothing in the codebase...#2021-04-0108:58nilernIt never comes up in input validation and our DbC story is not really there yet. Whereas Spec was basically created for DbC and third party tools are needed for most other things.#2021-03-3013:12Adam HelinsHmmm, took me a while to realise that was a source of error and I cannot seem to find how to merge such maps:
[:merge
[:map [:a :int]]
[:and
[:map [:b int]]
[:fn some-pred]]]
The problem is that the second map is enclosed by [:and] , and Malli doesn't seem to understand that it is ultimately a map, meaning that :merge does not behave like it is merging 2 maps.
How can I merge them while keeping that custom predicate?#2021-03-3014:30nilernhttps://github.com/metosin/malli/pull/405 (but still unreleased)#2021-03-3019:20Adam HelinsExcellent, once again @U055NJ5CC saves the day#2021-03-3113:46ikitommishipped in 0.4.0 now#2021-03-3105:23valeraukoAfter seeing the #Performance section of malli's readme I'm quite interested in trying it out. One thing I was wondering is how it fares at compile-time? I'm just assuming that it achieves high runtime speed by offloading as much as possible to macros. Are there any numbers how this affects eg lein startup time?#2021-03-3105:29ikitommi@vale good question. There are no macros in malli (ok, one mandatory). the “schema compile time” is at runtime, you can have a separate step to create an immutable & optimized validator/transformer/parser and reuse that.#2021-03-3105:30ikitommiin many cases, even without reuse, it’s still faster than with alternatives, e.g. with spec / spec-tools - one-shot “gimme a validator, use it once and forget it”.#2021-03-3108:11valeraukoawesome, time to dive in then! thanks for the great work!#2021-03-3105:32ikitommimalli.core (without sci) should load under 1sec from source and 0.1sec from classes.#2021-03-3105:34ikitommi(= int? (m/validator [:and int? int? [:and int? [:and int?]]]))
; => true
#2021-03-3105:46ikitommi(= identity
(m/decoder
[:map
[:id :int]
[:name :string]
[:tags [:vector :string]]]
(mt/json-transformer)))
; => true#2021-03-3107:24robert-stuttafordwas hoping that humanize would use language like 'at least one string? is required' here, but instead, i get "unknown error". am i doing it wrong, or is this a TODO for humanize?
(m/explain [:sequential {:min 1} string?] ())
;; {:schema [:sequential {:min 1} string?]
;; :value ()
;; :errors (#Error{:path []
;; :in []
;; :schema [:sequential {:min 1} string?]
;; :value ()
;; :type :malli.core/limits})}
(-> (m/explain [:sequential {:min 1} string?] ())
me/humanize)
;; ["unknown error"]#2021-03-3107:46ikitommi@robert-stuttaford just missing an error mapping for :malli.core/limits. PR welcome to malli.error.#2021-03-3107:46ikitommiyou can also play with the mappings by passing them into me/humanize:
(-> (m/explain [:sequential {:min 1} string?] ())
(me/humanize {:errors {::m/limits {:error/message {:en "there's no limits"}}}}))
; => ["there's no limits"]#2021-03-3107:52ikitommiuncomplete, but close:
(defn min-max-sequence-error-message [error options]
(str ((me/-pred-min-max-error-fn {:pred identity}) error options) " elements"))
(-> (m/explain [:sequential {:min 1} string?] ())
(me/humanize {:errors {::m/limits {:error/fn {:en min-max-sequence-error-message}}}}))
; => ["should be at least 1 elements"]#2021-03-3108:05Adam HelinsThe following scheme systematically produces a stack overflow. It is a recursive definition where one item (`:E`) contains another item (which might be :E as well, or not).
I am surprised this stack overflow is that systematic. There must be something wrong somewhere unless I am missing the obvious. Tried with :or instead of :multi, same thing.
(def registry
{:A [:tuple [:= :A]]
:B [:tuple [:= :B]]
:C [:tuple [:= :C]]
:D [:tuple [:= :D]]
:E [:tuple
[:= :E]
:item]
:item [:multi
{:dispatch first}
[:A :A]
[:B :B]
[:C :C]
[:D :D]
[:E :E]]})
(malli.gen/generate [:schema
{:registry registry}
:item])#2021-04-0108:33nilernThere was https://github.com/metosin/malli/issues/359 also#2021-03-3108:07Adam HelinsExecution error (StackOverflowError) at malli.core/-schema (core.cljc:242).
#2021-03-3108:11Adam HelinsValidation results in the same, I probably should open an issue actually#2021-03-3108:11valeraukoawesome, time to dive in then! thanks for the great work!#2021-03-3108:32ikitommi@adam678 would [:ref :item] work? It's a lazy reference type#2021-03-3108:40Adam HelinsIn the :E definition? Such as:
[:tuple
[:= :E]
[:ref :item]]
I've tried it different ways but get an exception: :malli.core/invalid-ref {:ref :item}
Sorry, seems I didn't drink enough coffee today 😅#2021-03-3108:41Adam HelinsRight, refs must be namespaced...#2021-03-3109:10Adam Helins@ikitommi However, there is still something wrong with generation. Adding another recursive item makes things very slow (eg. 2-3 seconds for generating something like [:E [:E [:A]]].
Adding a third recursive item makes things so slow that my fans are churning and after a couple of minutes there still isn't any result.#2021-03-3109:26ikitommiplease write an issue, malli should lean on https://github.com/clojure/test.check/blob/master/doc/intro.md#recursive-generators, but it does not.#2021-03-3109:45Adam HelinsI did: https://github.com/metosin/malli/issues/408
I hope I am not spamming. It's just I don't quite remember having that sort of problems with Spec so I am having a bit of a hard time working with that.
Not complaining though, so far using Malli has been very comfortable!#2021-04-0108:02ikitommihow about lazy generators? https://github.com/metosin/malli/pull/409#2021-03-3108:33ikitommithere is no recursion detection for eager references.#2021-03-3108:34ikitommiI Think StackOverflow is a correct way to fail here#2021-04-0108:35nilernIt makes sense but does leave something to be desired#2021-03-3108:58fugbixGreetings everyone!! Thank so much for this super neat lib, really cool and useful.
I have a bit of an issue, whereby requiring malli.core triggers a ClassNotFoundException:
nREPL server started on port 55188 on host 127.0.0.1 -
REPL-y 0.4.4, nREPL 0.8.3
Clojure 1.10.3
Java HotSpot(TM) 64-Bit Server VM 11.0.9+7-LTS
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
user=> (require '[malli.core :as m])
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:471).
malli.impl.util.SchemaError
user=>
my project is leiningen and includes many dependencies (including borkdude/sci because I am using multi dispatching, Datomic, Kafka etc). I am using the latest version of malli (`metosin/malli "0.3.1"`), Of course using malli from a toy lean leiningen project works perfectly fine.
Although not a Clojure expert, I’ve never seen something similar. And I really can’t imagine why malli.impl.util doesn’t load?
Any help would me much appreciated, I’ll keep digging!#2021-04-0108:40nilernNo idea, I don't see anything nonstandard going on there#2021-04-0112:45fugbixThank you @U4MB6UKDL I have removed my target directory and everything’s fine now. Of course I would love to understand what brought it in this state, but I’ll carry on with work for now. Sorry for the fuss, and thanks again for that wicked malli lib, absolutely loving it.#2021-03-3109:08robert-stuttafordwonderful thank you @ikitommi!#2021-03-3109:26dharriganA little while ago, there was a conversation about stripping nulls from the return back to the client#2021-03-3109:26dharriganhjmmm...let me think a bit more...#2021-03-3109:42dharriganSo, okay, I'm doing this in the setup of the ring/router#2021-03-3109:42dharrigan:muuntaja (m/create
(assoc-in m/default-options
[:formats "application/json" :opts]
{:mapper (-> (j/object-mapper)
(.setSerializationInclusion JsonInclude$Include/NON_EMPTY))}))#2021-03-3109:43dharriganFollowing throught the code, I believe I'm handed back an instance of the ObjectMapper from jsonista.core when I then set that option#2021-03-3109:43dharriganthe thing is, if I do this (now!), then all of my coercion fails (using malli)#2021-03-3109:43dharrigani.e.,#2021-03-3109:43dharrigan"coercion": "malli",
"errors": [
{
"in": [
"repId"
],
"message": "missing required key",
"path": [
"repId"
],
....
....#2021-03-3109:44dharriganif I put it back to :muuntaja m/instance, then all is well with the world#2021-03-3109:44dharriganI'm a bit lost as to why it fails#2021-03-3109:45dharriganit feels like it's not able to parse the incoming JSON data from the request#2021-03-3110:37dharriganI realise this is in the wrong channel (maybe I should put into #reitit)#2021-03-3113:45ikitommipushed out [metosin/malli "0.4.0"]. It’s a new MINOR as the :multi parse is changed (fixed) and leaning on the old (broken) code can break someone. https://cljdoc.org/d/metosin/malli/0.4.0/doc/changelog#2021-04-0108:33nilernThere was https://github.com/metosin/malli/issues/359 also#2021-04-0108:19ikitommiWelcome lazy generators! recursive generators are now orders of magnitude faster, ping @adam678
(time
(let [schema [:schema {:registry {::A [:tuple [:= :A]]
::B [:tuple [:= :B]]
::C [:tuple [:= :C]]
::D [:tuple [:= :D]]
::E [:tuple [:= :E] [:ref ::item]]
::F [:tuple [:= :F] [:ref ::item]]
::G [:tuple [:= :G] [:ref ::item]]
::item [:multi {:dispatch first}
[:A ::A]
[:B ::B]
[:C ::C]
[:D ::D]
[:E ::E]
[:F ::F]
[:G ::G]]}}
::E]
valid? (m/validator schema)]
(is (every? valid? (mg/sample schema {:size 10000})))))
; => "Elapsed time: 230 msecs"#2021-04-0108:27Adam HelinsThat was fast! Just tested it with my real world example and it seems to work just perfectly. (Generating WASM instructions where an instruction may contain a list of instructions).
Thanks a lot!#2021-04-0207:44Adam HelinsI am wondering how Malli is performing in an inheritance-like scenario. Essentially, Malli allows keys to be :optional in :map . This is about having whole sets of keys being optional, in an all-or-none fashion, such as:
(def AnimalBase
[:map ...])
(def Cat
[:merge
AnimalBase
[:map ...]])
(def Dog
[:merge
AnimalBase
[:map ...]])
(def Animal
(m/schema [:multi {:dispatch :type-or-something} :cat Cat :dog Dog]))
Does it leverage the fact that the AnimalBase schema is mandatory in all dispatched schemas?
Is there maybe an already existing better way of representing inheritance? Also assuming that being able to use generation is mandatory.#2021-04-0208:09juhoteperi:merge with two map schemas creates a new map schema with keys from both schemas, so Cat and Dog don't refer to AnimalBase but are instead just map schemas with properties from the inputs#2021-04-0208:09juhoteperiOr hmph, I might be confusing it with malli.util/merge... not sure if this works similarly#2021-04-0208:12juhoteperi:merge should same as malli.util/merge#2021-04-0208:12juhoteperihttps://github.com/metosin/malli/blob/master/src/malli/util.cljc#L401#2021-04-0208:13Adam HelinsI believe malli.util/merge is indeed eager in that sense (creating a new map schema and forgetting about AnimalBase). But theoretically, since this example relies on data and :merge, that :multischema could deduce that all dispatched schemas are maps that could have a common set of keys.
But it is just the most straightforward way of representing single inheritance IMO, maybe there is something better.#2021-04-0208:35Adam HelinsSorry I realized my example was a bit misleading so I deleted it so it doesn't spam the channel. I'll work more on this and try to come back with a real-world one.#2021-04-0310:55Setzer22Hi there! At our startup we're considering moving from spec to malli for our next project. I'm particularly excited about the clj-kondo integration support. Is there somewhere I could read more about it? Perhaps some of you using it? Did anyone integrate this into their repl/editor workflow (so that annotations are regenerated every time a new schema definition is made)? I'd also like to know how "deep" the static analysis goes. Is it just for elemental types like int or string? Or does the malli+clj-kondo combination understand your own compound types to some degree?#2021-04-0511:24ikitommihttps://twitter.com/puredanger/status/1378850899091578881#2021-04-0513:54emccueRelevant to the above, it feels like you could make spec out of malli but not the other way around#2021-04-0518:21nilernThere is more in the box for sure. On the other hand it's good that Spec does not have all that because it is in core#2021-04-0518:38emccueHow much of that "in the box"-ness is just about having a single maven artifact and how much is actual interdependency though?#2021-04-0519:00nilernAt least in theory we could just have the schema introspection and everything else could use that and live in separate jars#2021-04-0519:02nilernBut currently there is a monolithic schema protocol which does set a lower bound on the box size#2021-04-0519:05nilernSee also https://clojure.atlassian.net/browse/CLJ-2251 but I recall Tommi saying that walking Specs would still be janky#2021-04-0519:20marciolThere is spec/visitor in spec-tools @nilern. I think that the real problem is that it's impossible to decode serialized specs in runtime.#2021-04-0519:21marciolYou can go only one way without a proper way to reverse#2021-04-0519:29nilernYes the ticket mentions spec-tools and various details that I know little about since i never touched spec-tools#2021-04-0616:48ribeloIs it possible to create a key in a map using decode?#2021-04-0616:49ribelosomething like {:a 1 :b 2} => {:c 3}#2021-04-0617:27nilernSure it is#2021-04-0617:31nilern(m/decode [:map {:decode/abc (fn [m]
(if (and (map? m) (contains? m :a) (contains? m :b))
(-> m
(dissoc :a :b)
(assoc :c (+ (:a m) (:b m))))
m))}
[:a :int] [:b :int]]
{:a 1, :b 2}
(mt/transformer {:name :abc}))
=> {:c 3}#2021-04-0617:34nilernhttps://github.com/metosin/malli#value-transformation does have this info although the explanations are a bit sparse#2021-04-0618:07ribeloI've read it, but somehow it didn't occur to me that I could use decode/encode on the whole map and not on an individual key#2021-04-0618:07ribelogreat!#2021-04-0618:09nilernYes that is the "key" to changing keys :drum_with_drumsticks:#2021-04-0618:43ikitommi👍 small nitpick: the example could be encode as it transforms the value out of schema. Or the schema should not require :a & :b as they are not in the result.#2021-04-0618:52nilernIndeed#2021-04-0713:44armedHey, have issue with malli.error/humanize, here is minimal example (cljs):
(let [Schema2 [:map
[:field keyword?]]
Schema [:map
[:foo int?]
[:bar [:or nil? Schema2]]]]
(malli.error/humanize
(malli/explain Schema {:foo 1
:bar {:field "test"}})))
Throws error: #object[Error Error: Vector's key for assoc must be a number.]#2021-04-0713:46armedWhat I'm missing here?#2021-04-0713:48armedBy the way, in clojure I get another error for same snippet:
Execution error (ClassCastException) at malli.error/-ensure (error.cljc:124).
class clojure.lang.Keyword cannot be cast to class java.lang.Number (clojure.lang.Keyword is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')
#2021-04-0713:50ikitommi@armed it’s a known issue, see https://github.com/metosin/malli/pull/333#2021-04-0713:50ikitommiyou could say [:maybe Schema2] and it should work.#2021-04-0713:51ikitommibut, need to be fixed to be always robust.#2021-04-0713:52armed@ikitommi thanks for explanation, going to use :maybe#2021-04-0715:22danielnealWhen I call malli.util/update on a schema, the properties of the key I update are dropped, is there any way to preserve them?#2021-04-0715:24danielneal(malli.util/update
(malli.core/schema
[:map
[:a {:optional true}
int?]
[:b {:optional true} int?]])
:a identity)#2021-04-0715:24danielneal;=> [:map [:a int?] [:b {:optional true} int?]]#2021-04-0715:26nilernLooks like a bug to me :thinking_face:#2021-04-0715:27danielnealShall I open an issue?#2021-04-0715:35nilernthinking-face util/update is implemented by getting the value, then setting it
But it would be reasonable to expect that update preserves the properties even if get + assoc would not#2021-04-0715:37danielnealI agree#2021-04-0715:38nilernI think an issue would at the very least be a better place to discuss how that should work#2021-04-0715:44danielnealcool, added one here https://github.com/metosin/malli/issues/412#2021-04-0720:11HankstenbergWhat would be a good way to "cut" a data structure using a schema that it partially adheres to? To keep what conforms to the schema and discard the rest? The best approach I can think of is to use m/explain and then use the data from :errors to run an update-in on the data. Is there a better way to do that? It seems to me like there should be.#2021-04-0720:21nilernSounds like a more general version of strip-extra-keys-transformer#2021-04-0720:23nilernOr perhaps the tentative coercer https://github.com/metosin/malli/issues/404#2021-04-0807:49ikitommi@roseneck if the strip-extra-keys-transformer is not enough (just map keys), please provide an example with schema, input and expected output.#2021-04-0812:56jeremieHi all, just wonder if there is a malli schema for validating reitit route?
like the reitit.spec namespace in reitit.core but for malli. My use case is describing API endpoints and referencing the reitit route for each endpoint would be great. Thanks.#2021-04-0813:37ikitomminot yet. One of the initial reasons for creating malli was to have a tool where one can deep-merge nested schemas without tears, to be used in reitir route data validation.#2021-04-0813:37ikitommiwould you be interested in a PR?#2021-04-0816:56Setzer22Hi there! Missing spec's validation now that I've migrated to malli, I rolled my own thing: https://github.com/setzer22/malli-instrument Let me know what you think! 😄#2021-04-0817:29borkdudeI wonder, didn't malli already have something to instrument functions with malli schemas? if not so, how were the function specs intended to be used?#2021-04-0817:38Setzer22If there is, I couldn't find it anywhere 🤷 What's there, besides function schema syntax, is a global function registry and a macro, m/=> to add functions to that registry. I just built a super simple thing on top of all this#2021-04-0817:41ikitommi@setzer22 looks good. Plan is to have the instrumentation in malli core lib, but just not there yet, see: https://github.com/metosin/malli/issues/349#2021-04-0817:41ikitommiAlso related: https://github.com/metosin/malli/issues/369#2021-04-1412:20Ian FernandezI've made some questions asking about Intrument impl#2021-04-1412:21Ian Fernandezwhat malli expects for it#2021-04-1413:11ikitomminoticed, thanks! need to think it over before answering.#2021-04-0817:45ikitommi... but the idea seems exactly what you have already implemented. Maybe the lib could be just a new ns under malli?#2021-04-0819:30Setzer22@ikitommi Oh, I'd be glad to contribute if you think it fits! 👍#2021-04-0819:35Setzer22let me know if you'd like me to submit a PR with this#2021-04-0910:57ikitommi@setzer22 I’ll draft my vision to the issue and then it’s all yours if you have time, planning to integrate the normal defn schema inferrer into it, needs var filtering etc. But, busy atm, I’ll link your lib to Malli README meanwhile, so people can pick it up.#2021-04-0910:59ikitommihttps://github.com/metosin/malli/commit/e57e0f546c096b8cb39b6b0d83c0ed3f9d555552#2021-04-0913:03Setzer22cool! I'll keep an eye on that, definitely interested 😄#2021-04-0913:03Setzer22also thanks for mentioning my lib in the readme 👍#2021-04-1017:41ikitommiSome breaking changes coming to the extender-api, moving -type and -type-properties from Schema to IntoSchema. And some new methods so that we can describe the Malli schemas with Malli:
(defprotocol IntoSchema
(-type [this] "returns type of the schema")
(-type-properties [this] "returns schema type properties")
(-properties-schema [this] "maybe returns :map schema describing schema properties")
(-children-schema [this] "maybe returns sequence schema describing schema children")
(-into-schema [this properties children options] "creates a new schema instance"))
#2021-04-1017:48ikitommihttps://github.com/metosin/malli/pull/414#2021-04-1018:01ikitommialso, much better print presentations now:
(m/-enum-schema)
; => #IntoSchema{:type :enum}
(m/schema [*1 "A" "B"])
; => [:enum "A" "B"]
(type *1)
; => :malli.core/schema#2021-04-1020:15HankstenbergIf I want to generate valid malli schemas of the form:
[:map
[:key1 int?]
[:key2 int?]
....
[:keyn int?]]
using another malli schema - a meta-schema so to speak - what would that meta-schema look like? I've tried:
[:tuple [:= :map] [:+ [:tuple simple-keyword? [:= :int]]]])
But it has a flaw: it e.g. generates:
[:map
[[:key1 int?]
[:key2 int?]
....
[:keyn int?]]]
one pair of square brackets too much around the key-value-tuples. Does anybody know how to solve this using malli?#2021-04-1021:00ikitommi@roseneck it’s a sequence:
(mg/generate
[:cat {:gen/fmap vec}
[:= :map]
[:* [:tuple :keyword :int]]]
{:size 42, :seed 200})
;[:map
; [:IjPg2 -5908]
; [:E+:H9l* 287044292]
; [:*vCe._5z 10637946460]
; [:kLq67 -125]
; [:J9?D+FY4G 718]
; [:*_1*Rc+A -908]
; [:?qi 1523]
; [:x.+ -1]
; [:D4*5 -13]
; [:.C4 -80]
; [:r-A 3161894]
; [:rcC 1118]
; [:Q*._k*w8iG 194253522]
; [:o!dfA 21159377159]
; [:nx!h 973119]
; [:C:!Rw 109221]
; [:?N38.*3? -11854145]
; [:lb5Z?9V?_v* -42]
; [:i!M.l?D+ 2026049]
; [:++k43x -141680]
; [:U- -8]
; [:*x 1491554935632]
; [:heX2 1957851783]
; [:_PZ?q3+! -83]
; [:*H*x8N:.! 1452]
; [:k_Oy2+LXD -864224065]
; [:p 55]
; [:*o+1ul*5?? -1766412]
; [:W7:-61 -17174129950]
; [:qv79TTj_+ 21380662]
; [:h -30089]]#2021-04-1021:04HankstenbergThanks a lot! 🙂#2021-04-1210:16Ian FernandezHi, I've asked some questions at a malli issue to understand some design motivations 🙂
https://github.com/metosin/malli/issues/349#2021-04-1210:17Ian FernandezShould be good if anyone here could contribute 🙂#2021-04-1219:50jcfHi all! 👋
I'm looking into specifying higher-order functions, and wonder if there's a schema that resembles Clojure's ifn?.
I notice there's no ifn? predicate in the predicate-schemas (nor does fn? make an appearance), so I'm leaning towards defining something in user space (assuming there's nothing in Malli that I've missed?). I'm hesitant to do something like [:alt keyword? [:=> :cat any?]] because I'm creating a closed schema that would need to be extended for any IFn that might crop up in future… thinking of a custom IntoSchema or similar maybe.
Any tips are much appreciated!!#2021-04-1220:08jcfI can do this:
[:fn ifn?]
#2021-04-1220:08jcfI'm just not sure if that's a bad way to go. 🙂#2021-04-1304:15ikitommithat's the way to do it now.#2021-04-1310:23jcfThanks, @U055NJ5CC!#2021-04-1308:17Ben SlessIs it by design that schema transformations like mu/assoc don't play well with an unregistered schema?#2021-04-1309:06ikitomminot by design, could you repro if there is something that doesn’t work?#2021-04-1309:28Ben Sless(def Foo
[:map
[:a int?]])
(mu/assoc Foo :b ::bar)
Minimal example#2021-04-1309:47ikitommiyou should declare the ::bar so that is is visible, either:
1. override the default registry
2. pass the registry into Foo when creting it (it closes over the creation time registry)
3. pass the registry into mu/assoc
(def registry
(merge (m/default-schemas) {::bar int?}))
(def Foo
(m/schema
[:map
[:a int?]]
{:registry registry}))
(mu/assoc Foo :b ::bar)
;[:map
; [:a int?]
; [:b :user/bar]]
(mu/assoc [:map [:a int?]] :b ::bar {:registry registry})
;[:map
; [:a int?]
; [:b :user/bar]]#2021-04-1309:54Ben SlessI managed to get myself into this corner like so:
• wanted content dependent schema
• wanted to parametrize the schema (makes it extensible)
• figured out I'd do it by delaying registry building and schema compilation to run-time.
• With registry I need ::my-schema
• Can't transform anything with ::my-schema at compile time
I can create a placeholder registry for it but it seems like it would lead to errors down the line#2021-04-1309:57Ben SlessI'd be happy to adopt a better idea#2021-04-1310:00ikitommime too 🙂 spec partially checks the references eagerly, partially lazily (e.g. s/keys), malli is currently eager.#2021-04-1310:01ikitommithere is an internal escape hatch: :ref doesn’t check the reference if :malli.core/allow-invalid-refs option is truthy.#2021-04-1310:01ikitommiit is used with local registries, which can have… holes.#2021-04-1310:03ikitommi(m/validate
[:schema {:registry {::foo [:ref ::bar]}} ;; incomplete registry
[:tuple {:registry {::bar int?}}
::bar ::foo]]
[1 2])
; => true#2021-04-1310:06ikitommiideas welcome how to make this good.#2021-04-1310:12Ben SlessI don't know if it's good, but perhaps a :delay or :defer schema, which delays registry lookup to validation, with ample warning, care, etc.#2021-04-1310:18Ben SlessPerhaps even wrap it in a function which will always emit warnings when it's called, i.e.
(mu/assoc Foo :b (m/schema (m/defer ::bar)))
STDERR: Deferred Warning *at* - instances of deferred schema must be provided with a registry at run time!
You can also throw when instantiating an explainer, transformer, or validator from it, which is when you actually need the registry#2021-04-1310:21ikitommicould it be just [:ref {:lazy true} ::bar]?#2021-04-1310:21Ben SlessAh, laziness has to be explicit#2021-04-1310:21ikitommioh, ref’s are lazy already :thinking_face:#2021-04-1310:22Ben Slessyeah, this didn't work 🙂#2021-04-1310:22Ben Slesswe need lazier laziness#2021-04-1310:23ikitommitry (m/-lazy ::bar options)#2021-04-1310:24ikitommirefs resolve eager by default, but one can create lazy refs with that.#2021-04-1310:24ikitommi(let [-ref (or (and lazy (-memoize (fn [] (schema (mr/-schema (-registry options) ref) options))))
(if-let [s (mr/-schema (-registry options) ref)] (-memoize (fn [] (schema s options))))
(when-not allow-invalid-refs
(miu/-fail! ::invalid-ref {:type :ref, :ref ref})))#2021-04-1310:24Ben SlessCool, it worked 🙂#2021-04-1310:24Ben SlessAlways good to know some black magic#2021-04-1310:25ikitommicould make a version of that which doesn’t require the options.#2021-04-1310:25Ben SlessIt makes me wonder why [:ref {:lazy true} ::bar] didn't work#2021-04-1310:26ikitommiit’s a property of the IntoSchema, not Schema instance.#2021-04-1310:27ikitommiby design, all the IntoSchemas are crated using a function, which can take properties how the IntoSchema works. Easy to extend the system that way and DCE drops all the unneeded schemas.#2021-04-1310:27ikitommifor example, it’s reletively easy to create custom collection schema types:
(defn -collection-schema [{type :type fpred :pred, fempty :empty, fin :in :or {fin (fn [i _] i)} :as opts}] ...)#2021-04-1310:28ikitommi:ref has:
(defn -ref-schema
([]
(-ref-schema nil))
([{:keys [lazy type-properties] :as opts}] ...))#2021-04-1310:28ikitommibut, could lift the lazy into a :ref schema property too. so one can say [:ref {:lazy true} ::bar] as data.#2021-04-1310:29ikitommiif you need that, please write an issue.#2021-04-1310:32Ben SlessThank you!#2021-04-1310:33Ben SlessI wonder if I should settle for m/-lazy#2021-04-1310:33Ben SlessIf I should consider functions prefixed with - as implementation detail, then I'd say that I shouldn't and open that issue#2021-04-1310:34ikitommithings starging with - are ok to use: https://github.com/metosin/malli#alpha#2021-04-1310:38Ben Sless> might evolve during the alpha
That's a risk I'm willing to take. I think if m/-lazy develops in any direction it won't be one which will have friction with what I'm trying to do, on the contrary.
Thanks again for the help and guidance, you rock#2021-04-1309:30Ben SlessAnother issue I managed to stumble on, I defined a dependent schema like https://github.com/metosin/malli#content-dependent-simple-schema
It works well but throws when I pass it to reitit routes when the coercion is compiled#2021-04-1309:34Ben SlessGreat, now I'm unable to reproduce it 😞#2021-04-1309:47ikitommiyou should declare the ::bar so that is is visible, either:
1. override the default registry
2. pass the registry into Foo when creting it (it closes over the creation time registry)
3. pass the registry into mu/assoc
(def registry
(merge (m/default-schemas) {::bar int?}))
(def Foo
(m/schema
[:map
[:a int?]]
{:registry registry}))
(mu/assoc Foo :b ::bar)
;[:map
; [:a int?]
; [:b :user/bar]]
(mu/assoc [:map [:a int?]] :b ::bar {:registry registry})
;[:map
; [:a int?]
; [:b :user/bar]]#2021-04-1310:14yuhanAre there built-in functions to throw errors on invalid input? The plans for instrumentation in the above issue are nice, but I'm looking for something simple like spec/assert#2021-04-1310:38nilernSeems not#2021-04-1310:40nilernYou can always (assert (thingy-validator dada)) but the error is not so useful#2021-04-1310:43yuhanYeah, I wrote my own for now:
(defn malli-assert
([schema value]
(malli-assert schema value ""))
([schema value msg]
(when-not (malli/validate schema value)
(throw (ex-info (clojure.string/join "/n"
(cons msg
(flatten
(malli.error/humanize
(malli/explain schema value)))))
{:value value})))))
#2021-04-1310:44nilernMake a PR?#2021-04-1310:44yuhanIt's not ideal because humanize returns nested messages according to the path of the error, which I just flatten into a single string#2021-04-1310:45yuhanOk I'll submit an issue, just wanted to check if it was a design decision not to have an assert#2021-04-1310:47nilernMaybe AssertionError would be more appropriate :thinking_face:#2021-04-1310:50nilernAnd maybe use *assert* and make it a macro#2021-04-1310:50nilernSpec assert seems to use ex-info and a separate *assert* equivalent var#2021-04-1310:28jcfI have a follow up question from https://clojurians.slack.com/archives/CLDK6MFMK/p1618257034389400 regarding emitting configuration for clj-kondo to pick up (which is an awesome feature by the way!).
I have schematised the following code:
(m/=> hash-map-by
[:=> [:catn [:f [:fn ifn?]] [:coll coll?]] map?])
(defn hash-map-by
"Returns a map of the items in `coll` keyed by `f`."
[f coll]
(into {} (map (juxt f identity)) coll))
The function takes an arbitrary function, f, and a collection that will be converted into a map by applying f and identity to each item in the collection. Pretty standard stuff. 🙂
When I emit clj-kondo config with (mc/emit!), I get the following EDN:
{:lint-as #:malli.schema{defn schema.core/defn},
:linters {:type-mismatch {:namespaces {example.hash-map {hash-map-by {:arities {2 {:args [:fn :coll], :ret :map}}}}}}}}
Please note, the 2-arity args say :fn and :coll returning a :map which means I get linting issues with something like (hash-map-by :user/id [{:user/id 1} {:user/id 2}]).
Is this a bug worthy of a pull request or am I once again demonstrating my naivety? 🙈#2021-04-1310:32ikitommicurrently there is no way to override per schema instance how the clj-kondo works, but would be easy to add. also, having an ifn? schema built-in, it could have the correct clj-kondo type. interested in a PR?#2021-04-1310:32ikitommifor the latter that is.#2021-04-1310:32ikitommifor the first, for the second, something like:#2021-04-1310:33ikitommi[:fn {:clj-kondo/type :ifn} ifn?]#2021-04-1310:34jcfI'm very interested in implementing this as we'd need it to complete the replacement of clojure.spec with Malli in our codebase, I think.#2021-04-1310:34jcfI can create a PR for sure. 💯#2021-04-1311:00jcf@U055NJ5CC can I just clarify what you're thinking in terms of a PR, please?
I can add #'ifn? to the predicate-schemas and then these tests pass:
(testing "ifn schemas"
(let [schema (m/schema ifn?)]
(is (true? (m/validate schema (fn []))))
(is (true? (m/validate schema (constantly 1))))
(is (true? (m/validate schema :keyword)))
(is (true? (m/validate schema (reify clojure.lang.IFn
(invoke [_] "Invoked!")))))))
Is that what you had in mind when you mentioned having an ifn? schema built in?#2021-04-1311:09ikitommiyes, but also mappings for transformers, generators, json-schema, humanized errors and clj-kondo.#2021-04-1311:11jcfI'll take a look at implementing the full feature set for the ifn? domain. 👍#2021-04-1311:12jcfI'll not implement proper generation of interesting functions. Don't want to put us all out of a job. 😉#2021-04-1311:21jcfOh, I think I see what you mean. You want ifn? to be parameterised so you can schematize the args and return values…?#2021-04-1311:22jcfSo in the schema generator you can do something more than this:
(defmethod -schema-generator 'ifn? [_ _] (gen/return ::ifn))
#2021-04-1311:23jcfI'd need to generate a function that returns valid data given valid arguments.#2021-04-1311:23ikitommireally, why?#2021-04-1311:24ikitommithere is already :=> and :function which have proper input & output generators#2021-04-1311:24jcfBecause I thought that was what you wanted. 🙂#2021-04-1311:24jcfI think I misunderstood.#2021-04-1311:24ikitommino, just the simple thing, lke fn? but bit different 🙂#2021-04-1311:24jcfIf ifn? can remain a simple predicate, I should be able to have a first pass at a PR before the lunchtime walk. 🙂#2021-04-1311:24ikitommi👍#2021-04-1311:47jcfOne pull request with my stab at adding ifn? to Malli's list of supported predicates. Gotta go for a lunchtime stroll with the pup; I'll check in when I'm back in about an hour. 🙇
https://github.com/metosin/malli/pull/416#2021-04-1310:38nilernSeems not#2021-04-1311:47jcfOne pull request with my stab at adding ifn? to Malli's list of supported predicates. Gotta go for a lunchtime stroll with the pup; I'll check in when I'm back in about an hour. 🙇
https://github.com/metosin/malli/pull/416#2021-04-1318:58yuhanIs it possible for schemas to self-reference? Naively trying to make a recursive schema causes a stack overflow:
(malli/schema
::tree
{:registry (merge (malli/default-schemas)
{::node :int
::tree [:or
::node
[:tuple ::tree ::tree]]})})
#2021-04-1319:00ikitommi@qythium use the :ref, luke:
(mg/generate
(m/schema
::tree
{:registry (merge (m/default-schemas)
{::node :int
::tree [:or
::node
[:tuple [:ref ::tree] [:ref ::tree]]]})})
{:seed 3})
; => [[-26764 [[1 73136] [13307055 -1]]] [-381 [[-3 587742] -243724556]]]#2021-04-1319:01yuhanawesome, thanks!#2021-04-1319:28yuhanSo schema references only in :ref and :map have to be qualified keywords? Strings appear to work too, but not plain keywords#2021-04-1319:36yuhanSeems like it, I found malli.core/-reference? in the source code which checks for qualified-keyword? or string?, but this doesn't seem to be documented#2021-04-1319:31yuhanAlso it seems that the ::foo shorthand syntax doesn't work on the http://malli.io playground#2021-04-1319:37ikitommiyes, references should be qualified keywords or strings. http://malli.io … must be a sci-thing.#2021-04-1319:45yuhanOk, filed an issue on the http://malli.io github#2021-04-1319:49yuhanTrying to use unqualified keys as references tripped me up quite a bit at the beginning, since this requirement isn't documented and the error message just says :malli.core/invalid-schema#2021-04-1319:50ikitommidoc enhancement PRs are most welcome.#2021-04-1319:51ikitommi(the error keyword could be better here)#2021-04-1319:57yuhanI'll just file issues for now if that's ok - still in the early stages of experimenting with the library and not confident of writing docs#2021-04-1319:59yuhanAnother strange thing I encountered:
;; This works as a schema
[:map {:registry {::foo :int}}
::foo]
;; so does wrapping the keyword in a vector
[:map {:registry {::foo :int}}
[::foo]]
;; to pass it options
[:map {:registry {::foo :int}}
[::foo {:optional true}]]
;; These are ok too
[:map {:registry {::foo [:tuple :int :int]}}
::foo]
[:map {:registry {::foo [:tuple :int :int]}}
[::foo {:optional true}]]
;; But not this??
(malli/schema
[:map {:registry {::foo [:tuple :int :int]}}
[::foo]])
;; => Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:16).
;; :malli.core/invalid-schema {:schema [:tuple :int :int]}#2021-04-1404:27ikitommi@qythium looks like a bug in parsing entries.#2021-04-1404:27ikitommiweird thing that haven’t bumped into it before.#2021-04-1408:27Setzer22I was wondering, is there dedicated support in function schemas to handle things like (fn [a, b, & {:keys [c, d]})
I'd like to be able to do: (m/=> fn-name [:=> [:cat Foo, Bar, [:map?? [:c Baz] [:d Other]]]]#2021-04-1408:29Setzer22I couldn't find something to put in place of :map?? in my example#2021-04-1409:08nilernI don't think so
You could do [:cat Foo, Bar, [:* [:alt [:cat [:= :c] :any] [:cat [:= :d] :any]]]] (permits duplicates, but so does the fn itself)#2021-04-1409:10nilernIt would be good to have something more convenient especially with Clojure 1.11: https://clojure.org/news/2021/03/18/apis-serving-people-and-programs#2021-04-1409:13nilernLike [:cat Foo, Bar, [:alt [:* [:alt [:cat [:= :c] :any] [:cat [:= :d] :any]]] [:map [:c {:optional true} :any] [:d {:optional true} :any]]] or something can handle the 1.11 feature as well but who likes to write it even once#2021-04-1409:18nilernYou could write your own utility fn to generate that but really it should be built in to the fn schemas somehow#2021-04-1409:19ikitommimaybe: [:cat Foo, Bar, [:& [:c Baz] [:d Other]]]?#2021-04-1409:27nilernBut it does not do normal varargs so :& could be confusing. :&n?#2021-04-1617:35schmeeis there a way to use Malli to remove nil values in a map? :thinking_face:#2021-04-1620:34Leonid KorogodskiThe oddest thing. I have two projects. Both use malli 0.2.1 (according to `lein deps :tree`). But one of them accepts the syntax `[:map [:a string?] [:b string?]]` just fine, while the other throws an error on `(restart)`:
data-spec collection [] should be homogeneous, 3 values found
Any idea what could be the cause?#2021-04-1620:35borkdude@lkorogodski Can you try with lein clean and then run your project again?#2021-04-1620:35borkdudeYou might have some left-overs in your target folder or something#2021-04-1620:36Leonid KorogodskiSame result.#2021-04-1620:57borkdude@lkorogodski Please don't paste such long output into slack but rather use something like github gist#2021-04-1620:57borkdudeWhat happens when you do:
(require '[ :as io])
(io/resource "malli/core__init.class")
#2021-04-1620:58borkdudejust checking if this isn't an AOT-related problem#2021-04-1622:21Leonid KorogodskiOk, sorry. Just a moment.#2021-04-1622:27Leonid KorogodskinREPL server started on port 64815 on host 127.0.0.1 -
Connecting to local nREPL server...
Clojure 1.10.3
(require '[ :as io])
=> nil
(io/resource "malli/core__init.class")
=> nil
(require '[malli.core :as m])
=> nil
(m/validator [:map [:a string?] [:b string?]])
=>
#object[malli.core$_map_schema$reify$reify__21001$fn__21022
0x5361e937
"
That seems to work. But when I call (restart), it fails.
The usage for the router is as follows:
["/my-route" {:get {:summary "..."
:parameters {:header {:authorization string?}
:query [:map
[:a string?]
[:b string?]}
;; other things
}}]#2021-04-1622:33Leonid Korogodskimap? works in place of [:map ...] but doesn't check enough, of course.#2021-04-1702:23Leonid Korogodski(defn restart []
(mount.core/stop)
(mount.core/start))
#2021-04-1712:14ikitommi@lkorogodski in your example, there are both clojure specs (`:authorization)` and malli schemas (`:query)`. You should pick either one via :coercion option in the route data and use it in all the places. The error comes from spec-tools, so I believe you have coercion set to clojure.spec.#2021-04-1712:14ikitommithere are malli examples under examples . Hope this helps you to the right direction#2021-04-1712:17ikitommi@schmee sure, nothing built-in I guess, but you can add a custom decoder into the :map schema, reduce over the all map entries and remove nils.#2021-04-1712:18ikitommithere are similar imps in malli.transform for stripping out extra keys and transforming keys.#2021-04-1712:51Leonid KorogodskiThanks!#2021-04-1713:12ikitommilooking at GraphQL fragments, would be relatively easy to support those in malli. But not sure if that would be a good idea. Maybe not.#2021-04-1719:04eskosIIRC they're very close to Turing complete, so maybe not 🙂#2021-04-1719:06eskosThere's https://github.com/Vincit/venia which is data based so maybe have venia/Malli thingy instead? Would save you from implementing your own GraphQL parsing stuff as well.#2021-04-1713:13ikitommiwriting GraphQL<->Malli would be easier with that for sure.#2021-04-1906:57Hugh PowellI'm trying to create a schema for a CSV data structure with defined, case insensitive headers. It must also be able to generate a legal data structure. I'm using the same pattern as clojure.data.csv, a vector of vectors with the first vector representing the headers and the remaining vectors representing the rows. I can create a schema for the headers that checks validity fairly simply
(def headers
[:cat
[:re "(?i)header1"]
[:re "(?i)header2"]
...
[:re "(?i)headerN"]])
But test.chunk doesn't handle flags (the (?i) bit), so I don't have a generator.
My next guess is to create a schema that can match a given string case-insensitively and then a generator that will generate a randomly cased version of that string. I can write these two functions, but can't work out how to create a schema with them. Any thoughts or docs you can point me to?#2021-04-1907:49ikitommi@hugh336 maybe a custom generator? [:re {:gen/gen gen/alphanumric} "(?i)header1"]#2021-04-1909:02Hugh PowellAha! Awesome, thanks for the quick reply :thumbsup:#2021-04-1909:22ikitommihere a {:gen/elements ["header1"]} would be good, just data and always correct.#2021-04-1909:22ikitommihttps://github.com/metosin/malli#value-generation#2021-04-1914:31bartuka@ikitommi I also have several malli schemas that need to verify for [:fn (complement str/blank)] and I saw an open discussion from some years ago in Malli about adding validation of stripped strings to [:string] base schema#2021-04-1914:31bartukais this still desired?#2021-04-1914:32bartukanot some years.. last year, sorry. thought was 2019 https://github.com/metosin/malli/pull/205#2021-04-1914:40ikitommiis the [:string {:min 1}] bad?#2021-04-1914:40ikitommimaybe (def NonEmptyString [:string {:min 1}]) liike we would do in Schema?#2021-04-1914:42ikitommior:
[:map {:registry {::non-blank-string [:string {:min 1}]}}
[:name ::non-blank-string]
[:address [:map [:street ::non-blank-string]]]]#2021-04-1915:02bartuka(m/validate [:string {:min 1}] " ") ;; => true but I expect false if no blank is allowed#2021-04-1915:03bartukaI like the idea of using {:trim true :min 1 :max 99} .. looks very clean and we dont need to add custom registries#2021-04-2008:03SathiyaIs there an option to support blank string in :enum schema. :maybe supports nil but not blank values. currently i used [:or empty? [:enum "foo" "bar"]] to do this validation but is there a way to create a optional-enum registry that supports this scenario. The problem I'm facing using :or is I'm getting two error messages for each options. It would be better to have one message that says, value can be blank or either one of the valid values#2021-04-2008:07delaguardo[:enum "" "foo" "bar"] should work
https://malli.io/?value=1&schema=%5B%3Aenum%20%22%22%20%22foo%22%20%22bar%22%5D#2021-04-2008:08delaguardohowever error message will hide empty string#2021-04-2008:08delaguardo["should be either , foo or bar"]#2021-04-2010:24SathiyaThanks @U04V4KLKC#2021-04-2011:55anonimitorafHi guys, I came across https://github.com/metosin/malli/issues/125 when looking for ways to "instrument" functions.
My question is, how do you guys "instrument" functions with malli at the moment?#2021-04-2018:12Ivan FedorovHey folks 👋
Are you aware of any generators built on malli ?
Interested in crux pull vector, pathom resolvers, entity constructors, fulcro queries#2021-04-2021:03ikitommicommented on the instrument issue, sorry for the lag.#2021-04-2021:03ikitommihttps://github.com/metosin/malli/issues/349#issuecomment-823596409#2021-04-2106:38anonimitorafThanks for the update! Looks good!#2021-04-2021:04ikitommi@d.ian.b do you mean value generators or schema generators (providers)? Both would be cool. plan is to support EQL for slicing schemas at some point.#2021-04-2022:08refsetUhh, I think the wrong I.F. 🙂 /cc @U0A5V8ZR6#2021-04-2106:51Ivan FedorovI mean codegen. I want to describe malli schema and then have all the goodies (pull vectors, pathom resolvers, etc) generated.
I may end up doing it myself, just asking if you have seen anything like this already.#2021-04-2108:03ikitommidon’t think such things exist yet, go for it 🙂 the current malli lib has a lot of optional ns’s for external things (json schema, dot, etc), not sure where the limit should be - what optional things should be in the core lib (as optional namespaces) and which should be separate libs. At least EQL<->Malli could be in the core. Or then just redesign the code into a monorepo like reitit and make all the extras as separate modules.#2021-04-2202:59rutledgepaulvI don't mind things being in core if they don't require dependencies but if it's going to bloat the malli dep tree would prefer separate modules so I can keep my projects lean. Maybe that's an easy way to draw the line?#2021-04-2614:28Ian Fernandez@U28A9C90Q @U3Y18N0UC#2021-04-2615:47marciolTo me seems that at first glance it'd be better to separate all functionality that isn't essential to make Malli works in separated libs, but if we think about Malli as a full experience in terms of how to deal with schemas - and I hope that it became the best in class tool to work with external data, to such a point that people of other ecosystems really envy the Malli experience - I think that the full-batteries included is a better approach, so I really don't mind if all those niceties are included on Malli itself.#2021-04-2021:07ikitommioh, and merged 414, breaking change for the extender api. But schemas can now be described with schemas (just need to describe all schemas first :)), the new IntoSchema protocol:
(defprotocol IntoSchema
(-type [this] "returns type of the schema")
(-type-properties [this] "returns schema type properties")
(-properties-schema [this] "maybe returns :map schema describing schema properties")
(-children-schema [this] "maybe returns sequence schema describing schema children")
(-into-schema [this properties children options] "creates a new schema instance"))
#2021-04-2021:08ikitommialso, most malli.utilfunctions got bit faster as schema instances now have a direct handle to parent, no need to create new instances. which is nice.#2021-04-2021:09ikitommie.g. ´m/type` is looked via the parent:
(defn type
"Returns the Schema type."
([?schema]
(type ?schema nil))
([?schema options]
(-type (-parent (schema ?schema options)))))
#2021-04-2021:09ikitommiso, the public api remains the same.#2021-04-2108:03ikitommidon’t think such things exist yet, go for it 🙂 the current malli lib has a lot of optional ns’s for external things (json schema, dot, etc), not sure where the limit should be - what optional things should be in the core lib (as optional namespaces) and which should be separate libs. At least EQL<->Malli could be in the core. Or then just redesign the code into a monorepo like reitit and make all the extras as separate modules.#2021-04-2109:32nilernLet's not forget https://github.com/kwrooijen/gungnir#2021-04-2112:00andrea.crottiI'm just trying malli for the first time today, and it looks pretty great already#2021-04-2112:01andrea.crottione thing I didn't quite understand though, if I annotate a function with (m/=> ...
is there a way to actually force that check at runtime to make it fail when the schema doesn't match?
I only see that related with generating a clj-kondo config, which is great but I thought that run-time assertions would be also there, even if disabled by default maybe#2021-04-2112:06nilernhttps://github.com/metosin/malli/issues/349 but @setzer22 has a working implementation https://github.com/setzer22/malli-instrument#2021-04-2112:09Setzer22Yes, I've been using my implementation for a couple of weeks now 👍 The base features are working, but sometimes malli has trouble generating human readable descriptions of errors#2021-04-2112:09Setzer22if you give it a try, please report any issues!#2021-04-2112:14andrea.crottiah nice, yeah no rush was just wondering if I missed something from the docs#2021-04-2112:42ikitommi@setzer22 yes, there is an open PR about robust humanized errors. Few combinations that don’t work, but the idea to fix it seems legit.#2021-04-2112:54Setzer22@ikitommi Good to know!#2021-04-2113:15andrea.crottianother thing I noticed is that there isn't a way to just throw an exception if something fails to validate. I ended up with something like
(let [my-map ...
errors (m/explain schema my-map)]
(if (nil? errors) my-map (throw (ex-info "validation failed" errors)))
but isnt't there an easier way to do that?#2021-04-2113:22nilernWe should have an assert...#2021-04-2113:27nilernThere wasn't even an issue so I made one https://github.com/metosin/malli/issues/420#2021-04-2113:56nilernHmm hard to make an efficient assert that makes the validator and explainer behind the scenes. Especially Cljs would be tricky.#2021-04-2115:47andrea.crottimm maybe you don't always need the explainer#2021-04-2115:47andrea.crottiwould be nice I guess but not sure it needs to be the default#2021-04-2115:56nilernDropping the explainer does not make it any easier.#2021-04-2117:34yuhanThanks for creating the issue! it must have slipped my mind.
Do you mean caching a validator at the assert's compile time? I would be worried about the validator then going out of sync when you redefine the schema during dev time#2021-04-2117:38yuhanMaybe the *assert* dynamic var could have 3 levels:
• runtime validate (slowest but correct)
• compile-time validator (more efficient, use if schemas are fixed)
• compiles to nil#2021-04-2213:08nilernYeah I figured you just forgot.
And I did mean the assert macro would create the validator#2021-04-2213:10nilernIt is probably ok to not do that in dev mode although sometimes assertion overhead is too much even then#2021-04-2201:12ZaymonHey all. I might be missing something essential here but I don’t fundamentally understand why this won’t compile:
(defn longer-than-5? [x] (<= 5 (count x)))
(def MySchema [:map [:some-key [:and string? longer-than-5?]]])
; :malli.core/invalid-schema {:schema #function[example/longer-than-5?]}
Is there something special about string?, int? ect that makes them more than just a function from a -> bool ?#2021-04-2201:36ZaymonLooks like I need to specify the function with [:fn longer-than-5?] !#2021-04-2213:12nilernWhat is special is that they are in the m/predicate-schemas registry which is merged into the default registry#2021-04-2301:37Zaymon@U4MB6UKDL That’s interesting. Thank you#2021-04-2209:21Yevgeni TsodikovHi,
I think the strip-extra-keys-transformer removes mandatory fields if there are in an [:or ... ] vector:
(def my-schema [:and
[:map [:a int?]]
[:or
[:map [:b int?]]
[:map [:c int?]]]])
=> #'...
(m/validate my-schema {:a 1})
=> false
(m/validate my-schema {:a 1 :b 1 :c 1})
=> true
(m/decode my-schema {:a 1 :b 1 :c 1} (mt/transformer mt/strip-extra-keys-transformer))
=> {:a 1}
Did I configure the schema wrong?
How can I work around it?#2021-04-2209:25ikitommimaybe:
[:and
[:map
[:a int?]
[:b {:optional true} int?]
[:c {:optional true} int?]]
[:fn {:error/messag "only :a or :b is allowed"}
(fn [{:keys [b c]}] (not (and b c)))]]
or:
[:or
[:map
[:a int?]
[:b int?]]
[:map
[:a int?]
[:c int?]]]
would those work for you?#2021-04-2209:25Yevgeni TsodikovChecking, I want to see how they integrate with reitit ’s swagger docs#2021-04-2209:26ikitommiyou can run from repl:
(malli.swagger/transform
[:and
[:map
[:a int?]
[:b {:optional true} int?]
[:c {:optional true} int?]]
[:fn {:error/messag "only :a or :b is allowed"}
(fn [{:keys [b c]}] (not (and b c)))]])
to see what comes out.#2021-04-2209:27ikitommiswagger doesn’t support anyOf JSON Schema, so the latter will not show in swaggr docs.#2021-04-2209:27ikitommiopenapi has that, but not integrated into reiti.#2021-04-2210:26Yevgeni TsodikovI’ve used the first option, it worked for me.
Thanks!#2021-04-2209:51DosHi @ikitommi
I am facing interesting behavior with explain. I don't understand why order matters...#2021-04-2209:53Dos(def params-function?
(malli/schema
[:function
[:=> [:cat map? [:maybe map?] map?] map?]]
{::malli/function-checker malli.generator/function-checker}))
(def next-function?
(malli/schema
[:function
[:=> [:cat map? [:maybe map?] map?] keyword?]]
{::malli/function-checker malli.generator/function-checker}))#2021-04-2213:47ikitommi@U0D8J9T5K oh, that’s bad. will fix it.#2021-04-2214:52ikitommiFixed in master: https://github.com/metosin/malli/pull/421#2021-04-2417:50ikitommi:gen/schema … this might be useful:
(mg/sample [:any {:gen/schema :int}])
; => (0 -1 -2 1 0 0 18 1 -3 -1)
(mg/sample [:string {:gen/schema [:int {:gen/min 10, :gen/max 100}], :gen/fmap 'str}])
; => ("11" "11" "11" "10" "10" "10" "10" "13" "13" "66")#2021-04-2707:20Adam HelinsAfter WASM, I am now describing a Lisp using Malli and once again am doing heavy use of recursive generation. I'm probably gonna be spamming a few questions in the following days.
That kind of :set are always empty, am I doing something wrong with :ref? No such problem with :vector.
(malli.gen/sample [:set
{:registry {::foo :int}}
[:ref ::foo]])#2021-04-2711:44Adam HelinsIs there any design decision behind not generating NaN and Infinity?
(defmethod -schema-generator :double [schema options] (gen/double* (merge (-min-max schema options) {:infinite? false, :NaN? false})))#2021-04-2711:45Ivan FedorovAny good intro? I’m playing with schemas, and can’t see a formal definition of it in README
Currently I’m thinking that [:schema] schemas must have a :registry followed by a single root subschema reference.
I’ve watched the talk by Tommi Reiman from London Clojurians. It’s helpful, but I’m lacking some formal learning material.
Thanks in advance!#2021-04-2712:10Ivan FedorovI’m trying to understand the best way to describe the entity types of an app.
Probably it’s just a single big registry then and multiple (def schema:entity-type-n) using it.#2021-04-2718:48ikitommi@ognivo most malli-based projects I’ve seen use a mutable global registry with custom register function like spec (README has a guide how to set that up), registering looks like:
(register :user/id :int)
you can also just define schemas as var in schema style, using the default immutable regisitry:
(def Userid :int)
there was an Entities and Values guide in the README, but it was removed.#2021-04-2815:23Ben SlessIt's also possible to delay schema compilation and registry instantiation to start-up time.
I built a registry dynamically and injected it into a Component system, the registry itself contained content dependent schemas which depended on input arguments. It was fun. I might cajole my employer to let write something about it#2021-04-2816:52ikitommilooking forward to that!#2021-04-2811:39pithylessI’m reworking some error handling - especially dev vs prod. What’s the story with virhe? Is there a version of that coming soon? Or perhaps even some reflections and retrospectives that we could apply to our own work?#2021-04-2816:57ikitommi@pithyless there is a unpublished copy of reitit-error-pretty-printer in malli. should extract the common parts of malli + reitit and push out to virhe. to be honest, might be better if virhe was a community owned clj-commons library, not ours. there would be a lot to do with colors, themes, cljs-support etc, which needs work, which needs time, which we don’t have that much extra. happy to contribute code, ideas and requirements to that. For now: just copy the existing reitit thing if you want the looks.#2021-04-2817:16pithylessare you referring to this? https://github.com/metosin/reitit/blob/master/modules/reitit-dev/src/reitit/dev/pretty.cljc
It'd be great to read about the ideas and lessons learned on the virhe README if you find some free time to catch your breath. Either way, thanks for all the ceaseless work and FOSS publishing! 🙂#2021-04-2816:58ikitommiI’ll write the ideas and lessons learned to virhe README, soon. Might have time to push out first version before summer vacations.#2021-04-2817:01ikitommi🥳 pushed just out [metosin/malli "0.5.0"] - small fixes & improvements and the ability to describe Malli Schema Syntaxes with Malli. Small breaking changes for library extenders, more info in the CHANGELOG: https://github.com/metosin/malli/blob/master/CHANGELOG.md#050-2021-04-28#2021-04-2821:29eoliphantHi, dunno if I’m doing something wrong, but I’m having issues with local/custom registries and qualified keyword entries. Validation works fine, but getting properties, the original form back, etc don’t seem to work
(def registry*
(merge
(m/default-schemas)
{:testfoo [:string {:a :b}]
:test/bar [:string {:c :d}]
::testbaz [:string {:a :b}]}))
(m/properties :testfoo {:registry registry*})
; => {:a :b}
(m/properties ::testbaz {:registry registry*})
; => nil#2021-04-2904:47ikitommi@eoliphant What happens is:
• :testfoo is not a qualified key, it’s contents are inlined
• ::testbaz is a qualified = schema reference, instead of the inlined contents, you get the reference back. References are internally of type :malli.core/schema , which can also have properties. To get the actual schema behind the reference, you can m/deref the schema.
• analogy is to Vars in Clojure, e.g. you get #'inc (var) back, not inc (function value)
(def registry*
(merge
(m/default-schemas)
{:testfoo [:string {:a :b}]
:test/bar [:string {:c :d}]
::testbaz [:string {:a :b}]}))
(m/type :testfoo {:registry registry*})
; => :string
(m/type ::testbaz {:registry registry*})
; => :malli.core/schema
(-> ::testbaz
(m/schema {:registry registry*})
(m/deref))
; => [:string {:a :b}]
(-> ::testbaz
(m/schema {:registry registry*})
(m/deref)
(m/properties))
; => {:a :b}#2021-04-2904:49ikitommithat said, not happy how the references work atm, in most malli-based codebases I’ve seen, there is a lot of manual m/deref / m/deref-all calls to inline things. Also, the current behavior is not documented properly. Not sure what would be a correct way to handle these. Ideas welcome.#2021-04-2904:56ikitommicompared to spec, any lookup to a registry pulls the actual value, but when the reference is part of some other specs form, it’s kept as it is;
(require '[clojure.alpha.spec :as s])
(s/def ::kikka int?)
(s/form ::kikka)
; => clojure.core/int?
(s/form (s/tuple ::kikka))
; => (clojure.alpha.spec/tuple :user/kikka)
#2021-04-2905:00ikitommithis sums the current functionality:
(m/schema [:tuple :testfoo ::testbaz] {:registry registry*})
; => [:tuple [:string {:a :b}] :user/testbaz]#2021-04-2916:12eoliphantah it was just that? lol. yeah, for now i think just a bit in the docs, maybe in the areas that ref using qualified keys, custom/local registries would be a start just so folks are aware.
I just happened to catch Arne’s quick vid on Malli and data modeling, and so gonna just take his approach of my own ‘schema’ name space that limits the surface area, and I can just jack in handling this. will think about what might be a better approach longer term. it does ‘feel’ that something like (m/properties :test vs ::test …) should ‘just work’ from the average user/client’s perspective, returning the same kind of representation. maybe an ‘easy’ 🙂 namespace like reitit?#2021-04-2918:39ikitommineed to think what would be a good resolution for this. today, my 2 cents are to make it work like spec:
• by default defer eager references on m/schema, and add an option not to do that (used in form creation)
would be a breaking change.#2021-04-2918:41ikitommiunrelated, but, content-dependent collection schemas on coming up. not sure how useful, but possible:
(def List
(m/-collection-schema
(fn [properties [child]]
{:type 'List
:pred list?
:empty '()
:type-properties {:error/message "should be a list"
:gen/schema [:vector properties child]
:gen/fmap #(or (list* %) '())}})))
(m/validate [List :int] '(1 2))
; => true
(-> (m/explain [List :boolean] [1 2 3])
(me/humanize))
; => ["should be a list"]
(mg/sample [List {:gen/min 4, :gen/max 10} :int])
;((-1 0 -1 -1 -1 -1 -1 0)
; (-1 -1 -1 0 -1)
; (0 0 1 -2 -1)
; (-1 0 -1 1 -1 0 -4 -1)
; (-2 0 0 -2 1 0)
; (-2 7 2 3 2 -1)
; (-7 -3 -1 7 -1)
; (0 -12 0 8)
; (-5 25 0 -2 -5 1 -1 -25 -10 1)
; (53 221 0 15 1 -42))
(m/form [List :int])
; => [List :int]
(mu/assoc [List :int] 0 :boolean)
; => [List :boolean]#2021-04-3001:46pserranoHello. I'm looking forward to use malli in a project and will be actively learning about it in the next days 🙂#2021-04-3011:15ikitommihiccup:
[:schema
{:registry
{"Order" [:map
[:items [:vector [:enum "SIM" "SAM"]]]
[:delivery [:enum "letter" "email"]]]
"SimDeliveryRule" [:fn {:error/message "If the order only includes a SIM card then the delivery method may be "letter"."
:error/path [:delivery]}
'(fn [{:keys [items delivery]}]
(or (not= items ["SIM"])
(= delivery "letter")))]}}
[:and "Order" "SimDeliveryRule"]]
maps (maybe compact syntax via schema parsing):
{:type :schema
:registry {"Order" {:type :map
:entries [[:items {:type :vector
:items {:type :enum
:values ["SIM" "SAM"]}}]
[:delivery {:type :enum
:values ["letter" "email"]}]]}
"SimDeliveryRule" {:type :fn
:error/message "If the order only includes a SIM card then the delivery method may be "letter"."
:error/path [:delivery]
:value '(fn [{:keys [items delivery]}]
(or (not= items ["SIM"])
(= delivery "letter")))}}
:value {:type :and
:values ["Order" "SimDeliveryRule"]}}#2021-04-3012:00Yehonathan SharvitHello there!
Malli is great.
A question regarding function validation (a.k.a => ):
Is there a way to automatically validate that all function calls are made with proper args during development time?#2021-04-3012:58borkdude@viebel More info here:
https://clojurians.slack.com/archives/CLDK6MFMK/p1619006445487300#2021-04-3013:18Yehonathan SharvitThank you @borkdude!#2021-04-3013:19Yehonathan SharvitGiven that instrumentation is not yet there in malli, what’s the common use case for :=>?#2021-04-3013:19Yehonathan Sharvit“Only” generative testing?#2021-04-3013:20borkdudeIt's a good question, I've also wondered myself.#2021-04-3013:20borkdudeAt least you can generate some clj-kondo type hints with it =)#2021-04-3013:22Yehonathan SharvitI just tried clj-kondo type hints and it’s awesome.
A question related to that: where is the appropriate place for putting the (mc/emit!) function call?#2021-04-3013:22borkdudeMaybe in "component system du jour" start/reset?#2021-04-3013:23Yehonathan SharvitI didn’t know you were speaking french.
I don’t have any component: I am writing a library#2021-04-3013:24Yehonathan SharvitI am also wondering how the emitted config from the library is going to be integrated in the application that uses the lib#2021-04-3013:24borkdudeyou probably shouldn't do this in your library, it's something an end user should do#2021-04-3013:25borkdudeMaybe emit it as part of a pre-commit hook or something#2021-04-3013:25Yehonathan Sharvitwhy wouldn’t my library be responsible for emitting clj-kondo config for the function it provides?#2021-04-3013:26borkdudeYou can do that, but you should just commit that config into git and not generate it at runtime#2021-04-3013:26borkdudeclj-kondo has a mechanism to pick up on configs from libraries#2021-04-3013:27borkdudeDescribed here: https://github.com/clj-kondo/clj-kondo/blob/master/doc/config.md#exporting-and-importing-configuration#2021-04-3013:30Yehonathan SharvitNice!#2021-04-3013:51Yehonathan SharvitIs there already a script that generates clj-kondo config from project that uses malli (basically that calls (mc/emit!)?#2021-04-3013:51Yehonathan SharvitNot sure I could use babashka for that as my library might not be babashka compatible#2021-04-3013:52borkdudeMalli is not babashka compatible either, so you should use a JVM script / function for this. You could just write a function which you can call with clojure -X my.lib/gen-clj-kondo#2021-04-3013:52Yehonathan SharvitMalli is not babashka compatible. That is a sacrilege!#2021-04-3013:53borkdudeThere is an issue / petition to add malli to bb:
https://github.com/babashka/babashka/issues/737
Feel free to upvote.#2021-04-3014:01Yehonathan SharvitIt’s not an easy choice: What’s your opinion on that? Did you upvoted or downvoted @borkdude?#2021-04-3014:05Yehonathan SharvitAnother question related to malli’s :=> : Is there a way to automatically generate a doc string?#2021-04-3014:42ikitommi@viebel function instrumentation comes with this: https://github.com/metosin/malli/issues/349#2021-04-3014:44ikitommilot of things are still TODO, help welcome#2021-04-3014:55Yehonathan SharvitYeah. @borkdude pointed me to this Github issue.#2021-04-3014:55Yehonathan SharvitWe are starting to embrace malli at work. Hopefully, we’ll be motivated to help#2021-04-3014:56Yehonathan SharvitA question related to maps. Is it possible to specify that some combination of fields are forbidden?#2021-04-3014:56borkdude@viebel What made you choose malli over spec or both?#2021-04-3014:57Yehonathan Sharvit@ikitommi gave us a great talk about Malli at our dev meetup at work 😜#2021-04-3015:00Yehonathan SharvitWe chose malli mostly because:
1. In malli, schemas are data (not macro required)
2. spec seems stuck. Not sure if spec2 will be compatible with spec 1
3. It’s easier to contribute to malli than to spec
#2021-04-3015:05borkdudeInteresting, thanks for sharing. It seems there are two ways people are choosing libraries:
1) choose core unless ... , because: bundled with clojure (no additional deps), authority (core, cognitect), promoted as "the default"
2) choose community, easier to contribute / freedom, usually more focus/options than core libs#2021-04-3015:14Yehonathan SharvitDon’t forget the data driven aspect of malli!#2021-04-3015:23mynomotoHaving a version of spec alpha with a replacement in development for years doesn't exactly inspire confidence.#2021-04-3015:23borkdude@viebel In the defn podcast you argued that namespaced (fully qualified) keywords are a vital part of data oriented programming. spec promotes this by having specs bound to global keywords so specs describe meaning without context. Is this aspect sufficiently present in malli?#2021-04-3015:25borkdude(btw, I liked the podcast :))#2021-04-3015:29Yehonathan SharvitGood question @borkdude. I need to play more with Malli in order to answer this question.
(I am so glad you liked it)#2021-04-3015:39Yehonathan SharvitIs there a way to spec a map using a key from a registry but having the key non namespace qualified?#2021-04-3016:53ikitommi[:map {:registry {::id :int}
[:id ::id]]#2021-04-3017:59borkdude#2021-04-3017:59borkdudeThe work Jim is doing is maybe interesting for the authors of malli as well#2021-05-0215:03ikitommidefinitely, looks great @jimka.issy. Malli’s primary goal has been to be a data-driven runtime schema engine, but currently bending it for development time tooling to see how far we can go. Some wishes:
• a schema-base case-macro, with clj-kondo based error reporting on non-exhaustive matches - for things like :or and :multi. Genus seems to have those already?
• schematized fns. so many syntax options, the lisp curse?
• as soon as someone builds a pluggable intellisense / code completer for clojure, happy to emit malli-mappings so that one could complete known map keys (and get errors on invalid keys - via clj-kondo).#2021-05-0215:29ikitommicurrent status: https://twitter.com/ikitommi/status/1363753100268421122#2021-05-0215:03ikitommidefinitely, looks great @jimka.issy. Malli’s primary goal has been to be a data-driven runtime schema engine, but currently bending it for development time tooling to see how far we can go. Some wishes:
• a schema-base case-macro, with clj-kondo based error reporting on non-exhaustive matches - for things like :or and :multi. Genus seems to have those already?
• schematized fns. so many syntax options, the lisp curse?
• as soon as someone builds a pluggable intellisense / code completer for clojure, happy to emit malli-mappings so that one could complete known map keys (and get errors on invalid keys - via clj-kondo).#2021-05-0100:58Alexis VincentAre there any projects or examples out there for nifty crux/Malli compat stuff. Schemas or crux helper functions etc#2021-05-0110:17refsetGood timing! See @U0A5V8ZR6 's new lib https://mobile.twitter.com/spacegangster/status/1388092289084428299#2021-05-0110:22Ivan Fedorov@U051V5LLP is the initiator and the sponsor on this one 🙏#2021-05-0111:17Alexis VincentYeah so cool! I had a 5 min play with this yesterday! Great work!#2021-05-0111:18Alexis VincentAny further examples/projects anyone has seen? @U899JBRPF#2021-05-0112:12refsetNothing else that's hit my radar yet, but I'm also very interested by the intersection :) Semi-related: I am also curious about the possibility of Malli->Alloy tooling that could assist with modelling in Crux#2021-05-0113:12Alexis VincentCan connect in a bit and see what we come up with#2021-05-0114:20dvingo@UJWLUPW13 do you have any use-cases in mind you're looking for? or dev workflows? I have some of my own ideas for helper code I'd want related to crux, but I'm interested to hear of others.#2021-05-0114:21dvingo@U899JBRPF do you have any links or materials on alloy as it relates to crux?#2021-05-0114:32Alexis VincentNot yet. Just getting started integrating malli and crux (and new to both). Just wrote a transformer that santises crux entities to be passed on to a json api. transforms :crux.db/id -> id and strips out entity type im transacting in. Will get a better feel for what I need as I work with both#2021-05-0114:36Alexis VincentMalli coersion for crux database migration is going to be 👌#2021-05-0114:36Alexis VincentAnd UI generation#2021-05-0215:15Ivan FedorovMr Reiman, maybe you could be interested making a video review of the EQL generation I wrote and giving your opinion on a better way to write schema transformers? I think this could be beneficial for those writing transformers based on malli.
If yes — we could schedule a video call @U055NJ5CC
If no — no worries.#2021-05-0305:23ikitommisure @U0A5V8ZR6, would like to do that, just super busy at work atm, between projects. If the code - and the goal - is available somewhere, I could first try to read that, between things.#2021-05-0308:19Ivan FedorovYep, you can look at it here:
https://github.com/dvingo/malli-code-gen/blob/main/src/malli_code_gen/gen_eql.clj
We have a branch merge pending. If we merge it before you look at the code – we’ll leave you a link in
https://github.com/dvingo/malli-code-gen
cc @U055NJ5CC
Not rushing you with the review. If you can say at what period you will be probably available for that (in a week or three) that would help.
Good luck with the workload!#2021-05-0318:02refsetHey @U051V5LLP
> do you have any links or materials on alloy as it relates to crux?
Nothing published from me yet, but this is worth a look https://www.hillelwayne.com/post/formally-modeling-migrations/ and, whilst not directly Alloy-related, I think principles of https://en.wikipedia.org/wiki/Object-role_modeling are also relevant#2021-05-0323:27dvingoooohh this looks great, thanks for sharing! I have some reading to do 🙂#2021-05-0115:08Joeli'm mucking with jsonista/malli and would like to use keywords for some map values, eg:
{ :Nested { :Map { :enums "values" :other "string" }}}
How do i get the values for :enums to be keywords, eg :values... yet leave other strings as strings?#2021-05-0118:14ikitommi@joel380 you need to describe a schema with all the parts that need transformations. in your case, simplest way to do it would be:
(require '[malli.provider :as mp]
'[malli.core :as m]
'[malli.transform :as mt])
;; infer schema from example value
(def schema (mp/provide [{:Nested {:Map {:enums :values}}}]))
;; create json-decoder for the schema
(def from-json (m/decoder schema (mt/json-transformer)))
;; apply
(from-json {:Nested {:Map {:enums "values" :other "string"}}})
; => {:Nested {:Map {:enums :values, :other "string"}}}#2021-05-0203:59JoelThanks I got that working. I had a schema with [:enum :value1 :value2] I assumed that would do the conversion, but it didn't work until I added [:and keyword? [:enum ...]]#2021-05-0204:00Joelmakes sense though.#2021-05-0204:02JoelHow can I validate the length of strings?#2021-05-0206:08ikitommi[:string {:min 1, :max 10}]#2021-05-0218:44Joel@ikitommi How do i see the list of keywords I can specify? Is it to look at unit tests? For example, I didn't know where to find the min/max keywords.#2021-05-0305:25ikitommi@joel380 not atm. The new malli schemas of malli schemas feature allows us to describe the available properties formally as schemas, but it’s just the mechanism atm, no content shipped. Once all schmas are described (both properties and children), it will be THE documentation. For now, you need to read the sources. Documentation PRs always welcome.#2021-05-0305:29ikitommie.g.
;; currently
(m/properties-schema :string)
; => nil
;; after
(m/properties-schema :string)
; [:map
; [:min {:description "length must be at least"} :int]
; [:max {:description "length must be at most"} :int]]
… different schema applications (e.g. generation) will have their own overlays, which can be merged to get the full set of available keys. Should be easy to generate proper docs out of those.#2021-05-0305:30ikitommiafter the children & property schemas are done, one can also generate valid schema-hiccup out of a given registry 🙂#2021-05-0312:45Yehonathan SharvitIs there a way to forbid some combination of keys in a map?
For instance a map with either :a and :b or :c and :d#2021-05-0312:48borkdudeprobably using another predicate?#2021-05-0312:48borkdudeor using two different maps#2021-05-0312:51Yehonathan SharvitWhat do you mean by “using another predicate”?#2021-05-0312:53borkdude[:and [:map ...] [:fn (fn [...] ...)]]#2021-05-0312:54borkdudeSame as with clojure.spec#2021-05-0312:54borkdude(def my-schema
[:and
[:map
[:x int?]
[:y int?]]
[:fn (fn [{:keys [x y]}] (> x y))]])#2021-05-0312:55Ben SlessI was just going to ask if there's a way to specify constraints on relations between keys which does not involve defining an ad-hoc function#2021-05-0312:56borkdudeMaybe using two separate closed maps also works. I think it depends on your domain perhaps?#2021-05-0312:57Ben SlessLikely, although I was just thinking of a case like the one you illustrated, a constraint where the value at one key is greater than another#2021-05-0312:59Ben SlessThis pattern repeats so often, should it be a schema?#2021-05-0313:19Ben SlessSomething like:
(def preds {:> >, :>= >=, :< <, :<= <=, := =, :not= not=})
(defn -rel
[]
(m/-simple-schema
(fn [_ [pred a b]]
{:type :rel,
:pred (m/-safe-pred #((preds pred) (get % a) (get % b))),
:min 3,
:max 3})))
(m/validate
(m/schema
[:and
[:map
[:x :int]
[:y :int]]
[:rel :< :x :y]]
{:registry
(mr/composite-registry
m/default-registry
{:rel (-rel)})})
#_{:x 1 :y 2}
{:x 1 :y 1})
#2021-05-0313:23borkdudereminds me of jet :)
$ echo '{:a 1 :b 2}' | jet --query '(< :a :b)'
true
$ echo '{:a 1 :b 2}' | jet --query '(> :a :b)'
false#2021-05-0313:28ikitommiI think it’s good idea to experiment new relational schemas in the user space, both -simple-schema & -collection-schma are good ways to make these, like @ben.sless you demoed. The more stuff pushed into these data-languages, the more it looks like a simple sci.#2021-05-0313:29Ben SlessI also had some thoughts on the subject here
https://clojureverse.org/t/declarative-rules-for-relations-between-inputs/7623/5?u=bsless#2021-05-0313:30Ben SlessA big plus it could have over sci is getting the benefit of the JIT while still keeping the schema serializable#2021-05-0313:30ikitommionly 👍#2021-05-0313:31ikitommitrue, sci introduces a lot of overhead. and is big on cljs.#2021-05-0313:31borkdudemuch smaller than self-hosted CLJS though#2021-05-0313:32borkdudebut how many people are really using schema serialization?#2021-05-0313:32borkdudeI think most people are not#2021-05-0313:32ikitommii think so too.#2021-05-0313:32Ben SlessDoes select-keys translate to mu/get of all the keys then closing the schema?#2021-05-0313:33ikitommibut a lightweight map key dependency utility might be a good fit for many things. just (somone) needs (to invent) more syntax.#2021-05-0313:33Ben SlessThe schema serialization isn't my first (or third) priority here tbh, it's more about having more expressive schemas for the common 90% of cases and not having to define ad-hoc functions and errors for them, not to mention generators.#2021-05-0313:34ikitommihttps://github.com/metosin/malli/blob/master/src/malli/util.cljc#L233-L239#2021-05-0313:34ikitommihttps://github.com/metosin/malli/blob/master/src/malli/util.cljc#L233-L239#2021-05-0313:34borkdudeyeah, I think that's nice. I think clojure spec also doesn't have a good answer to this yet and it's pretty common#2021-05-0313:45Yehonathan SharvitIs there an answer to this in TypeScript?#2021-05-0313:49ikitommiI think you have to use union types with ts#2021-05-0314:11Yehonathan SharvitDoes it help to define constraints on map keys#2021-05-0314:11Yehonathan Sharvit?#2021-05-0317:39JoelI'm surprised that TypeScript is sophisticated enough to be used as inspiration, I guess I need to learn more. I've presumed Elm/Haskell are more sophisticated, eg.: https://www.youtube.com/watch?v=IcgmSRJHu_8#2021-05-1002:48mynomotoTypescript is way more pragmatic than elm/Haskell. Structural typing almost looks like malli schemas enforced at compile time.#2021-05-0313:29ikitommithere is also things like :select-keys -> https://malli.io/?value=%7B%3Aa%201%2C%20%3Ab%202%7D&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22ABCD%22%20%5B%3Amap%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aa%20%3Aany%5D%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Ab%20%3Aany%5D%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Ac%20%3Aany%5D%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Ad%20%3Aany%5D%5D%7D%7D%20%20%20%20%20%20%20%20%20%20%20%20%20%0A%20%5B%3Aor%20%0A%20%20%5B%3Aselect-keys%20%22ABCD%22%20%5B%3Aa%20%3Ab%5D%5D%0A%20%20%5B%3Aselect-keys%20%22ABCD%22%20%5B%3Ac%20%3Ad%5D%5D%5D%5D.#2021-05-0313:31ikitommiwould meander be great at describing the relations, as data?#2021-05-0313:36borkdudemalleander!#2021-05-0313:37ikitommiusing modified ben’s syntax:
[:and
[:map
[:a :any]
[:b :any]
[:c :any]
[:d :any]]
[:rel
[:or
[:and :a :b]
[:and :c :d]]]]
#2021-05-0313:38Ben SlessThe only objection I can come up with is that meander is too powerful, (i.e. the result will be hard to work with)#2021-05-0313:39Ben Sless"Here, have this fighter-jet fueled by liquid plutonium"
Scary#2021-05-0313:38borkdudeI think it can be confusing to overload the meaning of :or within :rel#2021-05-0313:39borkdudeMaybe just [:and [:map ....] [:or [:required-keys :a :b] [:required-keys :c :d]] [:gtk :a :b]]#2021-05-0313:39borkdude(gtk = greather than for keys, or something)#2021-05-0313:40Ben SlessThe rel syntax needs to be properly thought out, I just invented something ad-hoc to see if it could work#2021-05-0313:40borkdude[:keys/> :a :b]#2021-05-0313:40Ben SlessHow about :constraints?#2021-05-0313:40borkdude[:keys/require :a :b]#2021-05-0313:41Ben SlessThen constraints unify by default (like datalog) or disjoin when specified (via :or)#2021-05-0313:41borkdude[:k> :a :b], [:kreq :a :b]#2021-05-0313:42ikitommithis works already:
(m/validate
[:schema {:registry {::a :int
::b :int
::c :int
::d :int}}
[:or
[:map ::a ::b]
[:map ::c ::d]]]
{::a 1, ::b 2})
; => true#2021-05-0313:49Yehonathan SharvitBut it doesn’t scale. When you have other set of constraints you’d need to write down all the combinations of valid keys.#2021-05-0313:42borkdudeyeah, that's what I said in the start of the conversation: two disjunct map defs#2021-05-0313:42ikitommijust not with non-qualified keys (for no good reason)#2021-05-0313:42Ben Sless[:constraints
[:or
[:and
[:requires :a :b]
[:> :a :b]]
[:and
[:requires :c :d]
[:< :c :d]]]]
#2021-05-0313:42borkdudebut for rels between keys you could just have specialized ops like [:k> :a :b]: the :a key must be greater than the :b key, without introducing some new concept#2021-05-0313:43ikitommii kinda like it.#2021-05-0313:44Ben Slessspecial ops rub me the wrong way, for some reason. What's wrong with just :>?#2021-05-0313:44ikitommibut question is: from whom the declaration are for? schema writer? user? external docs?#2021-05-0313:45ikitommi(m/validate [:> 6] 4)
; => false
#2021-05-0313:45ikitommiit already exists#2021-05-0313:46Ben Slesshm, yes, but as a schema not an argument (if we go back to the [:rel x y z] or [:constraint ,,,] suggestion#2021-05-0313:47borkdude> What's wrong with just :>
That it means different things in different contexts, this can be confusing imo.#2021-05-0313:48borkdudeWhat if you want to really do numeric comparison like [:> :a 5] and 5 is also a key in a map?#2021-05-0313:49borkdudeIn jet I chose [:> :a #jet/lit 5]#2021-05-0313:49borkdudebut quickly this became tedious so I wrote a clojure interpreter#2021-05-0313:50Ben SlessThe most shorthand constraint syntax I can think of would be :!
An alternative which conforms to your suggestion would be :!/>#2021-05-0313:51Ben SlessDownside - it looks like Perl
Upside - concise and unique syntax#2021-05-0313:52borkdudeyou should always make a good trade-off between adding extra syntax (= complexity) and how much people are really going to use this#2021-05-0313:53ikitommi[:and
[:map
[:a :any]
[:b :any]
[:c :any]
[:d :any]]
[:keys/or
[:keys/and :a :b]
[:keys/and :c :d]]]
#2021-05-0313:53borkdudeyeah, I like that a lot better#2021-05-0313:54borkdudewhy even :keys/or, you can just use the normal :or here?#2021-05-0313:55ikitommitrue#2021-05-0313:54Yehonathan SharvitI was about to write the same thing#2021-05-0313:54Ben SlessThen adding a predicate function on the keys would look like
[:and
[:map
[:a :any]
[:b :any]
[:c :any]
[:d :any]]
[:or
[:and
[:keys/and :a :b]
[:keys/< :a :b]]
[:keys/and :c :d]]]
#2021-05-0313:54Yehonathan SharvitThe disjunction must not be about the keys#2021-05-0313:55Ben Sless:keys/or should mean "at least k1 or k2 should exist"#2021-05-0313:55borkdudewhat does :keys/and mean: both keys must be present? what does [:keys/and :a] mean then? Maybe [:keys/req :a] is a better name?#2021-05-0313:56Ben SlessMaybe even without a shorthand, keys/require#2021-05-0313:57Yehonathan Sharvit:keys/req doesn’t convey the fact that :a and :b are related#2021-05-0313:57borkdudebut what exactly does [:keys/and :a :b] mean then, how are these things related?#2021-05-0313:58Yehonathan SharvitI didn’t say I liked :keys/and#2021-05-0313:59Yehonathan SharvitMaybe [:keys/present :a :b]?#2021-05-0314:00borkdudebut you also didn't say what the semantic relationship between those keys is according to that op.
> :keys/req doesn’t convey the fact that :a and :b are related
:present also doesn't communicate a relationship, just as :required doesn't, it just means all those keys must be required/present#2021-05-0314:11Yehonathan SharvitI was wrong. There is no relationship between the keys.
But somehow I find it weird to say “required” inside an or condition#2021-05-0313:56Yehonathan SharvitAnd now, where would we put a custom predicate [:fn …]?#2021-05-0313:57borkdudeor :keys/with and :keys/without ;P#2021-05-0313:57ikitommioff to cook some food. good discussion, as always 👍#2021-05-0313:58Ben SlessThank you, feel like we hit on something useful and needed
Bon appetit!#2021-05-0314:00Yehonathan SharvitBy the way, my use case is real for a project at work.#2021-05-0314:00Ben Slesssame#2021-05-0314:00Yehonathan SharvitWe are writing a Hbase driver#2021-05-0314:00borkdudeI guess you can already express required keys like this as well? [:and [:map ...] :a :b :c] or do keywords not behave like predicates in malli?#2021-05-0314:01Ben Sless(m/schema [:and :a :b])
exception#2021-05-0314:02borkdudeok, [:and [:fn :a] [:fn :b]] ? :thinking_face:#2021-05-0314:02borkdude(ugly)#2021-05-0314:03Ben SlessAnd non informative, and useless for generation
(me/humanize (m/explain (m/schema [:and [:fn :a]]) {:b 1}))
;; => #:malli{:error ["unknown error"]}
#2021-05-0314:03borkdudeagreed#2021-05-0314:01Yehonathan SharvitThe scan function receives a map that could contain either :from and :to or :starts-with#2021-05-0314:01borkdudeIt looks like malli is becoming really popular in Israel?#2021-05-0314:02Yehonathan SharvitThere are not so many Clojure shops i Israel. Therefore, we could already say that 10% of Israeli Clojure shops use malli 😁#2021-05-0314:06Yehonathan SharvitI’m struggling to write the schema of the map received by scan
[:map
[:starts-with :string]
[:from :string]
[:to :string]
[:xor
[:keys/req :starts-with]
[:keys/req :from :to]
[:keys/req :from]
[:keys/req :to]
]
]
#2021-05-0314:06Yehonathan Sharvit:or is not good in my case. It should be exclusive or#2021-05-0314:07borkdudeif it is xor, then why not use disjunct maps?#2021-05-0314:07Yehonathan SharvitBecause there are other keys#2021-05-0314:07borkdudemaybe add a :type field or something?#2021-05-0314:07Yehonathan Sharvit:time-range , :limit …#2021-05-0314:08Yehonathan Sharvit[:map
[:starts-with :string]
[:from :string]
[:to :string]
[:limit :number]
[:time-range [:map [:from-ms :int] [:to-ms :int]]
[:xor
[:keys/req :starts-with]
[:keys/req :from :to]
[:keys/req :from]
[:keys/req :to]
]
]
#2021-05-0314:08Yehonathan SharvitWhat is a :type field?#2021-05-0314:09borkdudeI think you could have different types of ranges#2021-05-0314:11borkdude{:range-type :closed}, {:range-type :open}#2021-05-0314:11Ben SlessIn this case I'd start with some base schema, merge the different options with it and or between them#2021-05-0314:12Yehonathan SharvitI don’t get what both of you suggested#2021-05-0314:17borkdude@viebel I think this requirement is a bit weird:
:xor
[:keys/req :from :to]
[:keys/req :from]
if from is present, and to is not, then the second is true. but if to is present, then the first one is true. so to is optional.#2021-05-0314:17borkdudeI did not look at the other requirements, but I think your logic can be "refactored"#2021-05-0314:17borkdudeby modeling it differently#2021-05-0315:28Yehonathan SharvitI don’t see how to refactor the logic.
All the followings are valid
• :from 10 :to 20
• :form 10
• :to 20
• :starts-with aaa#2021-05-0315:28Yehonathan SharvitThe followings are invalid#2021-05-0315:29Yehonathan Sharvit• :from 10 :starts-with aaa
• :to 20 :starts-with aaa
• “from 10 :to 20 starts-with aaa#2021-05-0314:22emccuehmm, this is a tricky one#2021-05-0314:23emccue[:or [:map {:closed true}
[:from ...]
[:to ...]]
[:map {:closed true}
[:starts-with ...]]]#2021-05-0314:24emccueOR is an exclusive or if the types don't intersect#2021-05-0314:24emccue[:map
[:from ...]
[:to ...]]
#2021-05-0314:24emccuebut if you have it closed you can't have anything other than from and to#2021-05-0314:25emccueand if you have it open you leave open the possibility that :starts-with is in the map as well#2021-05-0314:25emccuewhat you need is basically#2021-05-0314:27emccue[:or [:and [:map
[:from ...]
[:to ...]]
[:not [:map
[:starts-with ...]]]]
[:and [:map
[:starts-with ...]]
[:not [:map
[:from ...]
[:to ...]]]]#2021-05-0314:27emccueOR#2021-05-0314:27emccue[:or [:and [:map
[:from ...]
[:to ...]]
[:not [:map
[:starts-with ...]]]]
[:and [:map
[:starts-with ...]]
[:not [:map [:from ...]]
[:not [:map [:to ...]]]]]
#2021-05-0314:28emccuenot that not exists, but conceptually thats what you would need to represent the constraint#2021-05-0314:28emccuesince you only want to be closed against specific keys#2021-05-0314:29emccueif you are willing to explicitly enumerate all the keys you can just do the version with {:closed#2021-05-0314:29Ben SlessI meant something like
(def Base
[:map
[:limit :number]
[:time-range [:map [:from-ms :int] [:to-ms :int]]]])
(def S1 (mu/merge Base [:map [:starts-with :string]]))
(def S2 (mu/merge Base [:map [:from :string] [:to :string]]))
#2021-05-0315:02Ben SlessDid some generalizing work, still not set on the syntax
(defn comparator-relation
[sym msg]
(let [f @(resolve sym)
type (keyword "!" (name sym))]
[type
(m/-simple-schema
(fn [_ [a b]]
(let [fa #(get % a)
fb #(get % b)]
{:type type
:pred (m/-safe-pred #(f (fa %) (fb %))),
:type-properties
{:error/fn
{:en (fn [{:keys [schema value]} _]
(str
"value at key "
a ", "
(fa value)
", should be "
msg
" value at key "
b
", "
(fb value)))}}
:min 2,
:max 2})))]))
(defn -comparator-relation-schemas
[]
(into
{}
(map (fn [[sym msg]] (comparator-relation sym msg)))
[['> "greater than"]
['>= "greater than or equal to"]
['= "equal to"]
['== "equal to"]
['<= "lesser than or equal to"]
['< "lesser than"]]))
(me/humanize
(m/explain
(m/schema
[:and
[:map
[:x :int]
[:y :int]]
[:!/> :x :y]]
{:registry
(mr/composite-registry
m/default-registry
(-comparator-relation-schemas))})
{:x 1 :y 1}))
#2021-05-0406:24Ben SlessNext step is figuring out how to derive generators from this#2021-05-0407:14Ben SlessMost cases just work besides equality#2021-05-0408:28Ben SlessHere we go:
(defn- derive-from-fmap
[schema options gen]
(let [props (merge (m/type-properties schema)
(m/properties schema))]
(when-some [fmap (:gen/fmap props)]
(gen/fmap (m/eval fmap (or options (m/options schema)))
gen))))
(defmethod mg/-schema-generator :and [schema options]
(let [[h & t] (m/children schema options)
base-gen (mg/generator h options)
gen (reduce (fn [gen schema]
(if-let [gen (derive-from-fmap schema options gen)]
gen
gen))
base-gen
t)]
(gen/such-that (m/validator schema options) gen 100)))
#2021-05-0410:59Ben SlessI collected the proof of concept implementation with some explanations and cleaned up code at a repo https://github.com/bsless/malli-keys-relations#2021-05-0412:53Ben SlessSome more thoughts:
• should references to paths inside maps be explicit rather than implicit? i.e. [:keys/> :x :y] vs [:keys/> [:path :x] [:path :y]] . The second syntax is more cumbersome but gives more freedom (`[:path :x :z]`) and facilitates more logic (see next points)
• Facts about collections. How do I say the value at key k1 must be contained in the collection in k2 , or a number smaller than the size of that collection? Stuff like [:contains? :x :y] (the value at y is in x), or [:> [:count :x] :y] (y is smaller than the count of x)#2021-05-0413:46dvingoVery cool stuff! I was playing with adding support for cljs yesterday. (resolve sym)(https://github.com/bsless/malli-keys-relations/blob/master/src/com/github/bsless/malli_keys_relations.clj#L112) is not allowed in cljs - I suspect converting this to a macro may get cljs support working.
In cljs resolve only allows a literal quoted symbol as an argument: https://cljs.github.io/api/cljs.core/resolve#2021-05-0414:07borkdudeNote that runtime resolve in general doesn't play well with GraalVM native-image unless it's executed at compile time#2021-05-0414:20Ben SlessI'll replace it with a map or defmulti#2021-05-0414:20Ben SlessI think defmulti will be best, extensible#2021-05-0415:28Ben Slessdone#2021-05-0415:29Ben SlessWhat about the alternative syntax?#2021-05-0506:48ikitommigreat work @ben.sless! need to read the code & think about this a bit. could you check the corresponding features from JSON Schema (https://json-schema.org/understanding-json-schema/reference/object.html#dependencies) so that Malli can be as compliant as it’s reasonable with it. I think your suggestion already a superset of what is in JSON Schema…#2021-05-0506:49ikitommimeanwhile, a quick poke on the schema inferring / providing:
• part1: make it suck less (5x faster on sample dataset): https://github.com/metosin/malli/pull/439
• part2: make inferring first class, most likely 100x faster: https://github.com/metosin/malli/pull/440#2021-05-0506:52ikitommiwith part2, all schemas are reponsible for describing how values can/not be described with schemas. This will open up things like inferring types from enums etc, e.g. [:enum "small" "medium" "large"] has a child schema of :string.#2021-05-0506:53ikitommithe part1 (5x faster) of new provider is now 68 loc, instead of the old 69. The old was kinda bloated anyways 😉#2021-05-0506:56ikitommi➜ malli git:(faster-inferrer-part1) ✗ cloc src --by-file
15 text files.
15 unique files.
0 files ignored.
v 1.86 T=0.03 s (571.3 files/s, 180498.1 lines/s)
----------------------------------------------------------------------------------------
File blank comment code
----------------------------------------------------------------------------------------
src/malli/core.cljc 146 46 1779
src/malli/impl/regex.cljc 93 23 508
src/malli/util.cljc 44 15 350
src/malli/transform.cljc 64 22 335
src/malli/generator.cljc 48 13 281
src/malli/error.cljc 24 6 249
src/malli/clj_kondo.cljc 21 3 141
src/malli/json_schema.cljc 23 3 128
src/malli/dot.cljc 7 3 69
src/malli/provider.cljc 10 3 68
src/malli/registry.cljc 15 3 65
src/malli/swagger.cljc 14 4 51
src/malli/impl/util.cljc 7 0 24
src/malli/edn.cljc 3 0 16
src/malli/sci.cljc 1 0 11
----------------------------------------------------------------------------------------
SUM: 520 144 4075
----------------------------------------------------------------------------------------#2021-05-0506:57ikitommi➜ malli git:(faster-inferrer-part1) cloc test --by-file
11 text files.
11 unique files.
0 files ignored.
v 1.86 T=0.04 s (264.8 files/s, 133395.9 lines/s)
-----------------------------------------------------------------------------------
File blank comment code
-----------------------------------------------------------------------------------
test/malli/core_test.cljc 330 6 1902
test/malli/util_test.cljc 115 4 789
test/malli/transform_test.cljc 93 3 714
test/malli/error_test.cljc 42 3 439
test/malli/generator_test.cljc 39 3 266
test/malli/swagger_test.cljc 11 4 258
test/malli/json_schema_test.cljc 14 4 236
test/malli/dot_test.cljc 12 0 73
test/malli/registry_test.cljc 10 2 61
test/malli/clj_kondo_test.cljc 8 0 55
test/malli/provider_test.cljc 4 0 41
-----------------------------------------------------------------------------------
SUM: 678 29 4834
-----------------------------------------------------------------------------------#2021-05-0507:26Ben SlessRegarding part 1, from the profiling I did yesterday it looks like most CPU is based on miu/-fail. I'll profile again with part1 merged to see what conclusions I can infer#2021-05-0508:14Ben Slessdisregard the last message I had some repl state#2021-05-0508:16Ben Slessok, see ~2x speedup#2021-05-0508:18Ben Sless-fail still dominates CPU, though.
Would you like me to prepare a MR for the exception speedup?#2021-05-0518:22Ben SlessQuestion about Malli's design - what's the rationale behind IntoSchemas? Why aren't Schemas and local bindings sufficient?#2021-05-0521:31ikitommi> For internal elegance, Malli is built using protocols.
>
It,s much easier to program and reason about the system with protocols than against functions / data. IntoSchema is analogous to a Class and Schena to an Object. There is one shared instance of :map IntoSchema (in a registry) which describes how to create :map Schema instances. And soon how to infer schemas from values. Kinda like adding static methods on classes, except it's all polymorphic.#2021-05-0612:19mike_ananevwhat is the difference between :or and :alt ?#2021-05-0612:22mike_ananevfound in docs, :alt is for spec inside seq#2021-05-0612:59mike_ananev@ikitommi how to define string constant in map spec? e.g version api is a constant#2021-05-0613:00mike_ananev[:enum "v1.0.0"] ?#2021-05-0613:00mike_ananevis there other ways?#2021-05-0613:02mike_ananevand why sets are not supported as spec definition? I cannot put
[:abc #{1 2 3}]
in map spec#2021-05-0613:11ikitommi[:= "v1.0.0"] or [:enum "v1.0.0"], both work.#2021-05-0613:14ikitommi#{1 2 3} doesn’t work as we didn’t want to reserver too much clojure syntax for special purposes. There is a hook to add support for that in the user space, but not documented as it’s not recommended.#2021-05-0613:15ikitommialso, now I think adding a shortcut syntax for regexps was not a good idea.#2021-05-0613:16ikitommiwhy? there is a cljs compiler warning about those, coudn’t fix it, instead of: #"kikka.*" would have been enough to have [:re #"kikka.*] or even [:string {:format "kikka.*"}]#2021-05-0613:18ikitommiI think in 90%+ cases people add anyway properties to the regexps like :error/message, so the benefit for supporting plain regexps is quite small.#2021-05-0613:25Ben SlessAn enhancement of regex error messages could be an indication at which character the match has failed.
"Should match regex" isn't terribly helpful for humanized errors#2021-05-1008:49nilernHost regexes don't support error positions. And I don't blame them, because the regex can fail in multiple ways at every character (our seqex schemas heuristically give the error the first longest partial match).#2021-05-1014:59Ben SlessIf we can do regex based generators we should theoretically be able to do regex based error reporting
No one said it would be easy, or even a good idea. You may not want to expose your regex via error messages#2021-05-0613:51mike_ananev[:= "v1.0.0"]👍#2021-05-0616:53JoelIs there a function that would give ":and" instead of ":or"? stricter vs. looser. maybe mu/intersect?
(mu/union [:map [:Event keyword?]] [:map [:Event [:enum :A :B]]])
=> [:map [Event [:or keyword? [:enum :A :B]...
#2021-05-0704:01ikitommiyou can configure the the mu/merge with few options, see`:merge-default` in mu/union https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L104-L113#2021-05-0704:03ikitommiWould that be mu/intercect with :and ? If so, PR welcome.#2021-05-0710:49ikitommi(my set theory skills are rusted on fridays)#2021-05-1008:52nilernIntersection would go with :and (which is a lattice meet)#2021-05-0618:50Ivan Fedorovany way to get schema name from RefSchema?
e.g. I have [:schema {:registry reg} ::task] and it’s already a RefSchema#2021-05-0619:22Ben Sless(m/deref schema)?#2021-05-0619:22Ben Sless(let [schema (schema ?schema options)]
(cond-> schema (satisfies? RefSchema schema) (-deref)))#2021-05-0619:23Ben SlessLooks like just what you need#2021-05-0619:43Ivan Fedorov@UK0810AQ2 thanks for the input! This gives something that looks like a keyword but in fact is a :malli.core/schema and I don’t understand how to get the keyword out#2021-05-0619:46Ben Slessah, hold on, let's dig some more#2021-05-0619:49Ben SlessWhat's wrong with just calling m/form?#2021-05-0619:53Ben SlessOkay, this is right:
(m/form (m/deref S))
You'll get back a keyword#2021-05-0619:54Ivan Fedorovwhat you see in m/children is still a schema!
but (-> m/form last) cuts it, thanks!#2021-05-0714:49Siddharth Jainis there a way I can set all map schema :closed by default? so that don't need to specify individually.#2021-05-0714:59dharriganI don't think there is, however, you could gather up all your schemas and map over them with malli.utils/closed-schema to recursively close them.#2021-05-0714:59dharriganThat may work#2021-05-0715:06dvingoYou could copy the -map-schema definition and make your own -closed-map-schema https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L616
so you could write:
[:closed-map [:xyz string?]]
not sure if that's worth it vs just typing {:closed true} though#2021-05-0715:17Siddharth Jainthanks guys, sounds good, cheers#2021-05-0716:10ikitommi@siddharthjain.in or better: create a PR that allows m/-map-schema to take :closed and then say:
(m/-map-schema {:naked-keys true, :closed true})
.. and you have a closed variant.#2021-05-0815:25cjsauerIs it possible to get malli to coerce blank strings "" into the :default value? Example:
(m/decode
[:map [:x {:default 0} int?]]
{:x ""}
(mt/transformer
mt/default-value-transformer
mt/string-transformer))
Should result in {:x 0}#2021-05-0815:34cjsauerEh, never mind. I’m thinking this isn’t a great idea. I can just strip empty strings from my data before sending it into malli for coercion. I think that’s clearer and less error-prone.#2021-05-0817:53dvingoI noticed that the the schema:
[:schema {:registry {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]}}
::cons]
doesn't work on http://malli.io (it doesn't produce DOT output or JSON schema)
while
[:schema
{:registry {"ConsCell" [:maybe [:tuple :int [:ref "ConsCell"]]]}}
"ConsCell"]
this one does work.
In my applications I would like to use fully qualified keyword schemas (RefSchemas) is there a way to get the transformers to work without walking the schema ahead of time and converting all fully qualified keywords to strings?#2021-05-0818:06ikitommi@danvingo it's a malli-sci-thing. I think one needs to define what is the current ns to sci so that ::cons works. Try :user/cons and it should work. I think @borkdude knows the answer how to make the :: work with sci...#2021-05-0818:14borkdude@ikitommi ::foo resolves to whatever the current namespace is, by default :user/foo#2021-05-0819:16dvingothanks Tommi - using :user/cons worked. I was also seeing some issues in a local repl regarding refschemas and dot. I'll try to get minimal repro#2021-05-0820:16borkdudeWelcome to xterm-sci.
user=> ::foo
:user/foo
user=>
https://babashka.org/xterm-sci/#2021-05-0820:16ikitommihmm... Might be a edamame thing?#2021-05-0901:11dvingoOk here's a small sample that I'm perplexed by:
(ns my-app.my-ns
(:require [malli.core :as m]
[malli.dot :as dot]))
(def comment-schema
{::id :uuid
::content :string
::replies [:vector [:ref ::comment]]
::updated-at inst?
::created-at inst?
::comment [:map
::id
::content
[::replies {:optional true}]
::updated-at
::created-at]})
(def reg {:registry (merge (m/default-schemas) comment-schema)})
(def c
{::id #uuid"ff43dad0-8007-47d7-845e-0b20a021bedb"
::content "hello"
::created-at (java.time.Instant/now)
::updated-at (java.time.Instant/now)})
(m/validate (m/schema ::comment reg) c)
; => true
(m/schema ::comment reg)
; => ::comment
I'm confused why this fails:
(m/schema [:schema reg ::comment] reg)
; =>
; Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:17).
; :malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1}
as well as these calls:
(dot/transform (m/schema ::comment reg))
; =>
; Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:17).
; :malli.core/invalid-schema {:schema ::comment}
;; trace:
malli.dot/transform dot.cljc: 58
malli.dot/transform dot.cljc: 60
malli.dot/-collect dot.cljc: 17
malli.core/walk core.cljc: 1655
malli.core/walk core.cljc: 1657
malli.core/-schema-schema/reify/reify/-walk core.cljc: 1276
malli.core/walk/reify/-outer core.cljc: 1662
malli.dot/-collect/fn dot.cljc: 21
malli.core/-properties-and-options core.cljc: 270
malli.core/-property-registry core.cljc: 265
clojure.core/reduce-kv core.clj: 6856
clojure.core.protocols/fn/G protocols.clj: 175
clojure.core/fn core.clj: 6845
...
malli.core/-property-registry/fn core.cljc: 265
malli.core/schema core.cljc: 1610
malli.core/-schema core.cljc: 251
malli.impl.util/-fail! util.cljc: 16
(dot/transform (m/schema ::comment reg) reg)
; =>
; Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:17).
; :malli.core/invalid-schema {:schema ::comment}
;; same trace as above
#2021-05-0902:22dvingoThis one also fails..
(def cons-schema {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]})
(def cons-list [1 [2 [3 [4 [5 nil]]]]])
(def reg {:registry (merge (m/default-schemas) cons-schema)})
(dot/transform (m/schema ::cons reg) reg)
; =>
; :malli.core/invalid-schema {:schema ::cons}#2021-05-0902:22dvingobut this works:
(m/validate (m/schema ::cons reg) cons-list) => true#2021-05-0922:59dvingoFollowing up on this, made some further investigations.
this works:
(dot/transform [:schema {:registry cons-schema} ::cons])
this fails:
(dot/transform [:schema {:registry (merge (m/default-schemas) cons-schema)} ::cons])
; =>
:malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1}
I'm not sure how adding the defaults would affect things. :
Related to this, what is the difference between these two?:
(m/schema [:schema {:registry cons-schema} ::cons]) ;; works
(m/schema ::cons {:registry cons-schema}) ;; fails -> :malli.core/invalid-schema {:schema :maybe}#2021-05-1121:28borkdude@ben.sless @viebel Hope you guys are ok... don't know where you live, but it seems some bad stuff was happening in your country today#2021-05-1207:54Ben SlessThank you 🙂
Been an interesting night, but we're all fine. Hopefully it will not escalate further#2021-05-1204:10Yehonathan SharvitThank you for asking @borkdude. The situation is tough. Kids don’t go to school. We are a bit scared. Hopefully, we’ll be fine.#2021-05-1307:38Adam HelinsIs there something in the current design that could prevent implementing that kind of schema salvaging? This is about altering some properties of an existing schema in a direct way:
{:int-vec [:vector :int]
:small-int-vec [:int-vec {:max 3}]}
I often have that kind of situation where I have a common schema (eg. :int-vec) but once in a while I have to put more constraints in properties or slightly alter generation (eg. {:max 3} ).
Is there another obvious way, besides an external function that recreates the schemas with optional properties?#2021-05-1307:55Adam Helins(and besides directly using malli.util)#2021-05-1311:50caumondHi everybody, I am looking for the documentation of :ref, I always have a doubt if it is needed or not. I find :ref in the example of the documentation but no sentence explaining why it necessary or not.#2021-05-1312:01ikitommi@adam678 interesting idea. Should the latter fail if it would have children too? Or swap those too, if present?#2021-05-1312:04ikitommi@caumond :ref is needed for recursion, no other utility I believe. It's implementation is lazy, so validators, explainers, generators etc. are realized only when needed.#2021-05-1312:04ikitommiDoc PR welcome#2021-05-1312:10caumond😁 I got it !#2021-05-1520:38caumondI posted a PR yesterday#2021-05-1312:07Adam Helins@ikitommi I was thinking of altering properties only since this is both useful and not controversial. Altering properties doesn't change fundamental aspects of a schema (usually, I guess).
If we could alter children, the following would look weird, almost evil:
{:int-vec [:vector :int]
:double-vec [:int-vec :double]}#2021-05-1410:08Adam Helins(I opened an issue since it looks like it's worth having a discussion: https://github.com/metosin/malli/issues/448)#2021-05-1411:11Adam HelinsDid you consider a :multi variant where the function directly returns a schema as opposed to a mere dispatch value? Like Spec does, actually. Once in a while I have to write something like:
[:multi {:dispatch first} [::a ::a] [::b ::b] [::c ::c] ...]#2021-05-1415:13snorremdHi. Is there a built in story for validating and coercing Record types? Using [:map [:fieldA :int] [:fieldB :int]] fails when Malli tries to humanise an error and calls the empty function in Clojure core on the record causing an error. Not sure if using the map schema is the correct way of going about this at all, or if it would be better to create a (malli.core/-simple-schema) or something. Any pointers would be appreciated.#2021-05-1710:16nilernI think :map matches the defrecord philosophy so the error you are getting would be a bug in error humanization. Humanization is not a fundamental operation so making records work should not be a big deal.#2021-05-1418:02Ivan FedorovI’ve gathered some observations about walking malli schemas to transform them into something (eql vector, clojure spec alpha).
Critique and additions welcome
https://github.com/dvingo/malli-code-gen/blob/main/src/dev/space/matterandvoid/transform_building_ideas.cljc
If anyone experienced with malli has 30 minutes – let’s record a youtube video together on writing malli code transformations. I think, this should be valuable for those who want to write their own transformers and consequently the community.#2021-05-1713:26eoliphantHi, i’m not 100% clear on how far down the rabbit hole you can go in terms of defining schemas in terms of other schemas. I’m using a mutable custom registry ‘spec’ishly’ per the docs, and i’m running into some issues, but not sure why. Perhaps some ‘type’ v ‘instance’ thing? For something like the following:
(register! :bigdec (m/simple-schema ...))
(register! :money [:map {:someprop ..}
[::amount :bigdec]
[::currency [:enum :USD :CAN ..]]])
(register! ::annual-income [:money {:propa ..}])
(def x {::amount 1.23M ::currency :USD})
(m/validate :money x)
;=> true
(m/validate ::annual-income x)
;Exception => :malli.core/invalid-schema {:schema [:map {:someprop ..} ....#2021-05-1716:53thomascothranQuestion about the schema walker functionality. I am trying to change all the :optional properties of a nested schema to false with this:
(mi/walk schema ;; malli.core aliased as `mi`
(mi/schema-walker
#(mu/update-properties % assoc :optional false)))
However, this only works if maps have a single key. If the schema is something like this:
(def schema
[:map
[:teams
[:vector
[:map
[:team
[:map
[:id uuid?]
[:name string?]
[:logoUrl {:optional true}
string?]]]
[:health {:optional true}
[:map
[:injuries {:optional true}
[:map
[:playerInjuries
[:vector
[:map
[:playerName string?]
[:position {:optional true}
string?]
[:status string?]]]]]]]]]])
Only only the values under the :teams key are updated; the :health key and all the values under it are unaffected. Am I using the schema walker incorrectly, or is this possibly a bug? This is on version 0.5.1
Edit: After looking at this more, I think it's just that update-properties doesn't work on keys, so I was using it incorrectly.#2021-05-1809:58Setzer22The syntax to define multi-arity fns is this one, right?
[:function
[:=> [:cat int? int?] int?]
[:=> [:cat int?] string?]]
#2021-05-1810:24Setzer22Ok, so I was asking because I'm implementing multi-arity schema support in my malli-instrument library, but I'm unsure how to best deal with instrumenting multi-arity fns with varargs. Due to the way in which varargs are specified, I don't have a way to know which of the arities under :function corresponds to each of the possible lengths of the arg list, or at least not without some analysis, consider the following:
[:function
[:=> [:* int?] int?]
[:=> [:* string?] string?]]
this is a valid schema for:
(defn foo
([x y z] (+ x y z))
([a b c d & _] (str a b c d)))
it can be argued that it's an innacurate schema for the function, but that's not my point: Regardless of the example, the fact remains that with seqex patterns being used to define arglists, one can't easily determine which one of the annotated arities to take given an arglist and its length :thinking_face:
The only way to do this with the current specification is to test each of the arity specs against the arglist until some validates. This is what I'm implementing now, but I thought it'd be good to bring out this discussion nontheless, in case you think the specification format could be improved 👍#2021-05-1811:08ikitommi@setzer22 please check https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L264-L272#2021-05-1811:09ikitommiand; https://github.com/metosin/malli/blob/master/src/malli/clj_kondo.cljc#L134-L154#2021-05-1811:12Setzer22I'm confused, what's this second one doing? 😅#2021-05-1811:10Setzer22nice, thanks! 👍 I was re-inventing the wheel for some of this work in a more inefficient way so this helps a lot#2021-05-1813:49Setzer22just noticed a small inconsistency when writing the instrumentation code:
(m/validate :cat nil) returns false, but when calling (fn [& args]) with zero arguments, args is bound to nil#2021-05-1813:51Setzer22I can easily work around it on my side using something like (or args '()), but I thought that may be something worth looking into#2021-05-1813:54Setzer22alright! malli-instrument now supports multi-arity fn schemas 😄 🎉
https://github.com/setzer22/malli-instrument
I'll continue working on this and pushing bugfixes#2021-05-2009:05Michaël SalihiHi!
Is Malli a good candidate to use for server side form validation in replacement for lib like https://funcool.github.io/struct/latest/? Especially thanks to malli.error / humanize?#2021-05-2009:10Michaël SalihiWith some hand made regex validators (email, etc) like https://github.com/funcool/struct/blob/master/src/struct/core.cljc#L195
+ locale support, I think so, right?#2021-05-2009:50ikitommi@admin055 definetely#2021-05-2009:54Michaël SalihiPerfect, thanks for the confirmation.
I'll toy with Malli in a project and see how to organize this. 👍#2021-05-2009:55Michaël Salihiis there an example demo project with form validation somewhere?#2021-05-2009:59Michaël SalihiI like very much the Reitit's example folder, if my tests are successful after a code review, I may be able to contribute with an example PR with form validation on Malli's Github.#2021-05-2009:59Michaël SalihiWhat do you think?#2021-05-2010:00ikitommisure, examples welcome! I have a malli-form (reagent) demo draft, but nothing serious.#2021-05-2010:02Michaël SalihiPerfect, let's do this!#2021-05-2010:07dharrigan has form validation#2021-05-2010:08dharrigan#2021-05-2010:08dharriganused here #2021-05-2010:10Michaël SalihiAwesome, thanks @dharrigan!#2021-05-2010:10dharriganyou're most welcome#2021-05-2016:18shanhi,
Anyone know if Malli has support that allows you to set a default value if the value doesn't pass the schema check?#2021-05-2016:20dharriganYou mean this? #2021-05-2016:33shanlooks like that just sets a default if the value is missing:
(m/parse
[:and {:default 42} int?]
"test");; => :malli.core/invalid
(m/decode [:and {:default 42} int?]
"test"
mt/default-value-transformer);; => "test"
I'll probably just use m/explain and do something based on if there are errors#2021-05-2104:49ikitommimight not be simple as any fix could change the conditional branch that should be used (with :or and :multi). But you should be able to use malli.util functions to access the invalid paths and work from there.#2021-05-2016:21shanpossibly. Generally I just want to check a map is valid but keep any good values and log out any errors#2021-05-2018:35Vladislavhi! Deref of [:merge ...] makes :registry with nested schemas of one of merged schemas invisible (but it still works). is it a bug, or feature, or there is some workaround Im missing?
point is - registry disappears from any of serialisations after deref, although it contains all information about recursive fields, and i assume there is no other way of define recursive schemas.
of course, i still could use underefed schema, but it much complicated#2021-05-2018:55ikitommiCan't recall where it was discussed, but we should clearly separate the locally registered schemas from normal registered ones and with the malli.util fns, expose the accumulated local registry, to be used in schema form. This makes the schemas visible and thus, serializable.#2021-05-2018:55ikitommismall change, clears thing a lot.#2021-05-2019:04VladislavI tried to find some way of extract and then serialize registry from schema (`m/-registry`, and then malli.registry/schemas
), but sadly unsuccess - it still needs more class serialization (may be to deref every schema from it? - leave it for tomorrow)#2021-05-2214:24Adam HelinsIf by any chance you ever had trouble with recursive data, please consider having a look at https://github.com/metosin/malli/issues/452
It is quite a hard problem and I would appreciate ideas, opinions, and reports if you had that kind of problems#2021-05-2218:09Adam Helins(and spec doesn't solve it either)#2021-05-2309:34Ben Slessis there a way of providing a transformer for key names? i.e. I expect a key :foo but after transformation the key will be :bar?#2021-05-2414:56ikitommi@ben.sless check out malli.transform/key-transformer#2021-05-2718:14Ben SlessI'm looking at JSON Schema now and trying to write it out with malli and recursive schemas (with the intention of writing a transformer from it to malli)
I'm using this BNF as reference
https://cswr.github.io/JsonSchema/spec/grammar/
I'm wondering what's the best schema to pick for modelling it
A lot of it is defined in terms of key-value pairs
I thought I'd try modelling it as a union
for convenience I defined
(defn -ref-u [& args] (into [:union] (map (fn [k] [:ref k])) args))
Then
;; JSDoc := { ( id, )? ( defs, )? JSch }
;; id := "id": "uri"
;; defs := "definitions": { kSch (, kSch)*}
;; kSch := kword: { JSch }
Becomes
::JSDoc (-ref-u ::id ::defs ::JSch)
::id [:map [:id {:optional true} [:ref ::uri]]]
::defs [:map [:definitions {:optional true} [:map-of :keyword [:ref ::JSch]]]]
But when it comes to n>=1 map schemas which are one-of types, I'm stumped
;; JSch := ( res (, res)*)
i.e., the map should have at least one res where it's defined as
;; res := type | strRes | numRes | arrRes | objRes | multRes | refSch | title | description
::res (-ref-u ::type ::strRes ::numRes ::arrRes ::objRes ::multRes ::refSch ::title ::description)
Ideally, I'd want some schema between map-of and map
I'm also not 100% clear on the difference between union and merge and if they're even the right choice.
Should I perhaps customize a -collection-schema?#2021-05-2915:02Ben Slessanyone?#2021-05-2719:11borkdudeIs this a standard? https://jsonlogic.com/#2021-05-2804:57ikitommihave looked at it, but haven’t heard of anyone using. not super popular.#2021-05-2804:57ikitommibut, I nice idea. not sure how many ways there are to describe such things.#2021-05-2915:02Ben Slessanyone?#2021-05-2807:50Pragyan TripathiIs there a way I can use malli to generate datomic schema?#2021-05-2820:53eoliphantnot ‘out of the box’ per se but it’s pretty straight forward. depends on the way you define your schemas. We use the dynamic registry sort of like spec, so our attributes are first class which sort naturally maps to datomic, and then just map over the registry for our attrs, :{db/ident <attr> :db/valueType (malli-to-datomic …) …}
theres of course specific stuff you’ll probably do for enums :my/enum [:enum :a :b] becomes norms for a :my/enum ref and :my.enum/a, etc. :map’s to refs, etc. We have our own custom options for additional customization when we generate.#2021-05-2820:56eoliphantalso, this video talks about this in general in terms of using malli to drive a lot of the rest of your app. https://www.youtube.com/watch?v=ww9yR_rbgQs&t=696s#2021-05-2821:05eoliphantis there a way to muck with option values based on a predicate? in pseudo-malli…
[:map
[:a :boolean]
[:b {:optional #(true? :a)} :boolean]]#2021-05-2909:19Adam HelinsWhy is :ref not allowed in seqex?
Are they truly incompatible or is it a goal implementing it later?#2021-05-2909:24ikitommiThere was a reason, @U4MB6UKDL should know thay#2021-06-0213:29nilernAllowing :ref would incidentally make context-free parsing possible, like in Spec. But that is much harder to specify and implement cleanly and efficiently e.g. I think left recursion drives Spec to a stack overflow. The implementation could be elegantly extended to do GLL parsing, like Instaparse. But supporting all CFG:s means supporting the ambiguous ones as well, so parse would have to return a seq of plausible parse trees etc. One could argue ambiguous schemas are undesirable anyway, but the best way to ban those is (some extension of) LR parsing and nobody wants to deal with shift-reduce conflicts either and many grammars are unambiguous but not LR (without manual mangling) 😩
I would say from a theoretical perspective there are some satisfactory options but it is not simply a matter of allowing :ref. From a UX standpoint it comes down to most programmers happily wielding regex but quickly getting confused with YACC (and even Instaparse when it comes to ambiguous grammars).#2021-06-0216:34Adam Helins@U4MB6UKDL That's a very thorough answer, thanks!#2021-05-2909:22ikitommi@eoliphant kinda like dependent schemas? I guess we could have a formal way of doing that, e.g. schema value -> schema mapping. There is already content-dependent schemas and some ways to do that at validation time#2021-05-2909:25ikitommiSee https://github.com/bsless/malli-keys-relations#2021-05-2909:26ikitommifor the simple key, :or or :multi would work too#2021-05-2909:26ikitommi1. ... or just :and with a :fn constraint#2021-05-2909:28ikitommibut with dependent schemas you could ask "what is the derived schema for this given value, against this (conditional) schema". Would be great.#2021-05-2909:28ikitommibut with dependent schemas you could ask "what is the derived schema for this given value, against this (conditional) schema". Would be great.#2021-05-2913:28eoliphantyeah i’d played around with :or and :multi, I guess :fn is the only route, was just trying to still leverage the ‘declared’ type#2021-05-3115:55respatializedApologies if I missed this while looking through the docs/source, but is there a way to make map schemas (and perhaps other schemas converted to the map syntax) self-documenting by adding something like:
(def my-map [:map [:k1 {:doc "A required key, typically used by the program to do foo."} :string] [:k2 {:optional true :doc "An optional key, typically used by the program for annotation purposes."} :string]])#2021-05-3116:02respatializedIt looks like malli.util/to-map-syntax and malli.util/from-map-syntax work just fine on the example I just gave, but I suppose I'm still wondering if the :doc kw in the props map is intended for a different purpose or potentially reserved in some way.#2021-05-3116:43ikitommi@afoltzm using qualified keys is the safest, plain keys could be used for something special, but not going to add anything special for :doc. So, safe in practice. :description is picked up JSON Schema transformer, so that works too and is reserved for docs.#2021-06-0106:00docoutoAny idea when the 'schema' branch could be merged into master?#2021-06-0106:01docoutoI mean, are there major things preventing this from happening?#2021-06-0114:51Ben SlessI hate to use the M word, but I think the answer to https://github.com/metosin/malli/issues/304 is yes. If we can say malli is working with types, aren't container schemas monads, of sorts? maybe is Maybe, or is Either, etc#2021-06-0115:03Ben Slessmap and and would be sort of product types?#2021-06-0213:35nilernI think Functor is more relevant than Monad here. tuple is a product too, or and some other things are sums. But still it is often not possible to jump to conclusions based on category or type theory.#2021-06-0213:37nilernI would rather have something like mu/fmap than select-keys automagically working through maybe etc.#2021-06-0213:39nilern(mu/fmap #(mu/select-keys % [:x]) [:maybe [:map [:x int?] [:y int?]]])
;=> [:maybe [:map [:x int?]]]
#2021-06-0214:49Ben SlessI agree with the observation, but I think the answer is the other way around - select-keys should work via fmap instead of knowing the implementation details of the container it's operating on#2021-06-0214:52Ben SlessAnd the reason I thought of Monads and not just Functors is cases like [:maybe [:maybe T]] which could be joined, and even hairier situations where you have some combinations of containers#2021-06-0214:54Ben SlessAnd if schemas are some representation of types, all the schema manipulations in malli.util could be represented as type manipulations which are implemented by the schemas instead of being bespoke data manipulations#2021-06-0707:14nilernI especially hate it when you run through several layers automagically, like in Cats I think (fmap inc [(just 3) 2]) ;=> [(just 4) 3] or something and JS Promises are not monads because .then tries to be both bind and (flipped) fmap...#2021-06-0707:22Ben Sless> anything automagically
I can't disagree with you there. These things should be explicit.
And yet, don't you think malli could benefit from abstracting schema manipulation and composition to interfaces of types manipulation and composition instead of data manipulation?
It lends itself to some weird phenomena, such as mu/get-in in a maybe-schema requires explicit reference to the path, so in [:maybe [:map [:a int?]]] int? is in [0 :a]. Is that the right way to go about it?#2021-06-0709:14nilernI haven't thought much about the schema manipulation stuff. But maybe the non-semantic (`[0 :a]` etc.) approach is not that bad and in fact kind of idiomatic. And I think even dependently typed languages like Idris will just pattern match on the type syntax even though it is much more against their philosophy (at least if the kind of the type is Type). It is hard to say what feels right when there is so little prior art (that I am aware of).#2021-06-0709:25Ben SlessThere's stuff over in typed land which also seems like leaky implementation and not a correct design choice
T1 = A | B
T2 = C | T1
Is not equivalent to
T3 = A | B | C
because unions are tagged, although it's still just a union of sets of input fields. They should be equivalent#2021-06-0711:16nilernIn a language with untagged unions those will be equivalent (or it is a bug). I don't really regard ADT:s as union types, especially since in most languages each variant can have multiple and even named fields.
And you can also do something similar with structural variant types (e.g. OCaml) although as usual type inference complicates that (unification must work etc.).#2021-06-0711:26nilernBy the way one reason lens types are hard is that not all containers have the contained things as type parameters e.g. data Point = Point {x : Int, y : Int} clearly contains Int:s but cannot be made a Functor or otherwise extract that fact on the type level because it's just an opaque name like :point#2021-06-0712:30Ben SlessYou mean it can be behind a reference?#2021-06-0809:03nilernIn SML it would be like a reference but in Haskell/OCaml/Java it is just an abstract type. Even If there was a way to get to the Int part(s) it is different from functors where it is always the first argument and you also have to decide between x, y or both...#2021-06-1802:35Ben Slesshttps://clojurians.slack.com/archives/CLDK6MFMK/p1623959758175200#2021-06-0306:13ikitommi@pedroabelleira need to clean it up first, hopefully before summer vacations#2021-06-0306:14ikitommia proposed fix to look up errors from parent schemas, finally: https://github.com/metosin/malli/pull/462#2021-06-0306:14ikitommifixes #86:
(-> [:map
[:foo {:error/message "entry-failure"} :int]]
(m/explain {:foo "1"})
(me/humanize {:resolve me/resolve-root-error-message-and-path}))
; => {:foo ["entry-failure"]}#2021-06-0306:15ikitommithe new hook in humanize allows custom collector, here, it just traverses the parents to look for error definitons, which will override the more exact definitions.#2021-06-0306:17ikitommiwe could have a m/humanizer that would prepare the humanization and would be much faster, e.g.
(def Schema
[:map
[:foo {:error/message "foo!"} :int]
[:bar {:error/message "bar!"} :int]])
(def humanize (me/humanizer Schema)
(-> Schema
(m/explain {:foo "1", :bar "1"})
humanize)
; => {:foo ["foo!"], :bar ["bar!]}#2021-06-0307:31Grigory ShepelevHello there. Need a little help. How do I malli/get-in with spec with registry? Suppose having the following structure
(def registry
{"users"
[:map
[:telegram
[:map
[:id int?
:is_bot boolean?]]]]
"messages"
[:map
[:from {:optional true} [:ref "users"]]
[:id int?]
[:text {:optional true} string?]
[:reply {:optional true} [:ref "messages"]]]})
And I've tried a lot of different combinations like:
(let [s (malli/schema [:schema {:registry registry} "messages"])
_ (print (malli/schema? s))]
(u/get-in s [:id]))
true;; => nil
No success.#2021-06-0308:27Ben SlessYou have three mistakes:
• syntax error in registry definition (look at the telegram map)
• the registry you pass to the schema constructor is incomplete, you need to merge it with the default registry
• The path you get-in is wrong, schema-schemas are considered part of the path and their children are at key 0.#2021-06-0312:38ikitommishould malli default to allowing the default registry o be swapped? e.g. strict mode where it can’t (for those who want to be fully in control).#2021-06-0314:29Ben SlessI think it would be better to merge the provided registry with the default registry be default. I always forget that#2021-06-0316:32ikitommione option should be not to have defauts schemas, only way to get proper DCE for tiny lib size, e.g. busy frontends.#2021-06-0316:33ikitommicurrently the smallest usefull malli-bundle is 2.3kb (gzipped js)#2021-06-0316:33ikitommiwith default registry, it’t 37kb.#2021-06-0317:50Ben SlessHow is that measured? I never worked with cljs. Does the compiler eliminate dead code?#2021-06-0613:56ikitommithere is a guide how to do that (with shadow-cljs) in malli readme. In short:
• unsed functions and protocol methods (and definitions) will be DCEd
• multimethods and deffed values are not#2021-06-0613:57ikitommithis is the reason all malli registry parts are behind a function -> no-one calling it => get’s eliminated under advanced.#2021-06-0614:30Ben Slesshuh, cool#2021-06-0312:40ikitommiseems that all the malli-codebases I have worked with, have introduced a custom mutable registry…#2021-06-0312:40ikitommiyou can always do evil:
(reset! @#'mr/registry* (mr/mutable-registry registry*))
#2021-06-0313:20ingesolHi! I want to create a schema for a sequence like this
[{:type :type1
:attrs {:type1-prop 1}}
{:type :type2
:attrs {:type2-prop 2}}]
:attrs schema will be different for different object types. There’s :multi, but the dispatch property :type is on the parent of the object inside :attrs.
Is the problem description clear, and does anyone have a good/best practice for solving this with malli?#2021-06-0315:56ikitommi@ingesol maybe:
(require '[malli.generator :as mg])
(mg/sample
[:multi {:dispatch :type}
[:type1 [:map
[:type [:= :type1]]
[:attrs [:map
[:type1-prop :int]]]]]
[:type2 [:map
[:type [:= :type2]]
[:attrs [:map
[:type2-prop :int]]]]]])
;({:type :type1, :attrs {:type1-prop -1}}
; {:type :type2, :attrs {:type2-prop -1}}
; {:type :type1, :attrs {:type1-prop -1}}
; {:type :type1, :attrs {:type1-prop -1}}
; {:type :type1, :attrs {:type1-prop -2}}
; {:type :type1, :attrs {:type1-prop 0}}
; {:type :type2, :attrs {:type2-prop 12}}
; {:type :type2, :attrs {:type2-prop 4}}
; {:type :type2, :attrs {:type2-prop -26}}
; {:type :type1, :attrs {:type1-prop -14}})#2021-06-0317:13ingesol@ikitommi Thanks! Yes, I thought about something like that too. My maps are bigger than in this minimal example, but easy enough to create the schemas with a factory function i guess.#2021-06-0319:36ChillPillzKillzBillzNewbie here... I am trying to run the base code example from the lambdaisland/regal example page https://github.com/lambdaisland/regal. My Deps.edn looks like {:deps {org.clojure/clojure {:mvn/version "1.10.3"}
`org.clojure/core.async {:mvn/version "1.3.618"}`
`org.clojure/test.check {:mvn/version "1.1.0"}`
`lambdaisland/regal {:mvn/version "0.0.97"}`
`metosin/reitit-malli {:mvn/version "0.4.2"}}` and my clojure code is as follows (ns baseClj.core
`(:require [malli.core :as m]`
`[malli.error :as me]`
`[malli.generator :as mg]`
`[lambdaisland.regal.malli :as regal-malli]`
`;; [lambdaisland.regal :as regal]`
`;; [lambdaisland.regal.generator :as regal-gen]`
`))` and first few lines (def malli-opts {:registry {:regal regal-malli/regal-schema}})
(def form [:+ "y"])
(def schema (m/schema [:regal form] malli-opts))
(m/form schema) my namespace definition gives me Error: "No namespace: lambdaisland.regal.malli"... but works when I remove the :as m . Then it fails on the lambdaisland.regal.malli, and again works when :as regal-malli is removed... Finally if fails at (def malli-opts {:registry {:regal lambdaisland.regal.malli/regal-schema}}) with the error ; Syntax error (ClassNotFoundException) compiling at (h:\Work\Clojure\baseClj\src\baseClj\core.clj:55:1).
; lambdaisland.regal.malli . None of this makes any sense... this is just the example code. What am I missing?#2021-06-0320:01ChillPillzKillzBillzI got a hint... The deps.edn include for malli was incorrect. I should have added metosin/malli {:mvn/version "0.5.1"} . This doesn't solve the problem tho...#2021-06-0323:05ribelohttps://github.com/metosin/malli/pull/305
is it dead or waiting for better times?#2021-06-0415:27ingesolI’ve been trying to get this to work, but cannot figure out what is going on. I’m expecting the :type value to be converted to a string:
(m/encode
[:schema {:registry
{::node [:multi {:dispatch :type
:decode/string #(update % :type keyword)}
[:malli.core/default
[:map
[:type :keyword]]]]}}
::node]
{:type :doc}
mt/string-transformer)
=> {:type :doc}
If I replace :malli.core/default with :doc, I get the expected string-coerced keyword
{:type "doc"}
Also, the schema has a local registry in order to be able to have nodes within nodes. If I remove the wrapping, the following also works:
(m/encode
[:map
[:type :keyword]]
{:type :doc}
mt/string-transformer)
=> {:type "doc"}#2021-06-0417:23datranWhat's the difference between parsing data and decoding data? My sense is that parse is similar to spec's conform, whereas decode is more about transformation - is that generally right?#2021-06-0417:39ikitommidecoding is a process of transforming values from external formats into valid clojure data. Parsing returns the parse trees.#2021-06-0417:44ikitommi@ingesol what version are you using. CHANGELOG says it's fixed in 0.5.0#2021-06-0417:44ikitommihttps://github.com/metosin/malli/issues/415#2021-06-0419:17ingesol@U055NJ5CC hmm, I thought I was on 0.5.2 or something, will check#2021-06-0419:23ingesolVery happy to be wrong, I was on 0.4.0. Thanks 🙂#2021-06-0419:29ingesolAaaand it works! Thanks for great help, as always#2021-06-1206:07robert-stuttafordis there a trick to debugging this m/explain exception? Vector's key for assoc must be a number. at https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L139#2021-06-1207:20ikitommihumaize should not throw, ever. There is an open issue/PR, seems stalled: https://github.com/metosin/malli/pull/333#discussion_r588855299#2021-06-1306:22robert-stuttafordi'll put together a repro and submit an issue. my problem seems different to the one that you have linked#2021-06-1314:46robert-stuttafordturns out i was overwriting instead of adding to the default error message set facepalm#2021-06-1215:08respatializedI feel like I may have asked this question before, but am having trouble finding the right keywords to search in the slack archive, so apologies for the redundant question: Does malli provide a meta-schema or helper function I can use to programatically validate forms that may or may not be valid malli schemas? Is there a self-describing malli schema for malli schemas?
Something like:
(m/validate malli-meta-schema [:a]) => false
(m/validate malli-meta-schema [:or [:fn int?] [:fn string?]]) => true#2021-06-1215:42ikitommi@afoltzm not yet. But since 0.5.0, each IntoSchema can describe it's properties and children schemas. Once all built-in schemas have described themselves (currently empty impls), one can ask from a registry the accumulated schema for schema AST.#2021-06-1215:47ikitommisee m/children-schema and m/properties-schema#2021-06-1216:06respatialized@ikitommi gotcha, thanks! right now I'm just doing a basic helper predicate that relies on the catching the exception thrown by calling m/schema on the form, which I think suffices for my current use case.#2021-06-1218:14borkdudeEnjoyed this podcast: https://twitter.com/JacekSchae/status/1402648727215542272#2021-06-1315:09robert-stuttafordis there a way to do this :enum {:error/fn entity.validation/malli-humanize-enum-error} for all enums in the spec, perhaps when calling m/humanize ?
otherwise i could walk the spec and inject them#2021-06-1317:12ikitommi@robert-stuttaford humanize takes an :errors option, where you can override :enum handling. Walking is another way to do it. There might be others too#2021-06-1318:25robert-stuttafordthanks @ikitommi - can you point me to example of the :errors option specifically overriding the printing of errors for a specific spec type like :enum that i can work from, please?#2021-06-1319:08ikitommisure @robert-stuttaford, https://github.com/metosin/malli/blob/master/test/malli/error_test.cljc#L177-L188#2021-06-1320:10robert-stuttafordah so:
(assoc-in me/default-errors [:enum :error/fn :en] my-fn-here)
#2021-06-1320:10robert-stuttafordthanks!#2021-06-1411:18ikitommiI would say (assoc me/default-errors :enum {:error/fn {:en my-fn-here}}) just to make sure that the whole :enum map is overridden, can’t recall which one is picked first, :error/fn or :error/message if both are present#2021-06-1421:43martinklepschHey 🙂 Is there a way to combine multiple schemas to one? Specifically I have a bunch of “base” keys and a few keys that I want to check based on the :type key in the set of “base” keys. I kind of see how I could do that with multi and a bit of repetition but I probably want :merge? Is there a way to get :merge without getting into registries and all that?#2021-06-1421:50martinklepschDuh, I just realized I can just into more key to existing [:map] schemas#2021-06-1504:45ikitommiyeah. :multi is not currntly mergable, oftenly asked. would allow concise definitions like:
[:merge
[:map [:id :uuid]]
[:multi {:dispatch :type}
[:pear [:size :int]]
[:boat [:price :int]]]]#2021-06-1615:03martinklepschIs there a “recommended way” to achieve this currently?#2021-06-1615:03martinklepschIt seems like a common need so if there is one I think documenting that could be useful. Happy to open a PR once I know 😄#2021-06-1507:07wcalderipeHey folks 👋
Why explain keeps passing values forward when previous operations in an and condition have already failed?
(m/explain
[:map
[:name
[:and string?
[:not {:error/message "non empty string"} empty?]]]]
{:name 1})
The example above throws Don't know how to create ISeq from: java.lang.Long. It does at (empty? 1). The conditions are concatenated with an and operator, so I wasn't expecting [:not {:error/message "non empty string"} empty?] to run because string? failed in the first place.
Note: there are probably better ways to express a non-empty string with Malli 😅#2021-06-1507:49Martín Varela[:string {:min 1}] is probably what you're after...#2021-06-1508:29ikitommiyes, I think it’s better to have keyword types (e.g. :string) with properties than to use the existing core functions, of which, many throw on invalid input.#2021-06-1508:29ikitommie.g. post-int? vs [:int {:min 1}]#2021-06-1511:46wcalderipe@U95NTJT4H thanks for the suggestion – I knew there was a better way to solve that problem.#2021-06-1511:47Martín VarelaGlad that helped 🙂#2021-06-1511:47wcalderipe@U055NJ5CC got it. For custom keyword types, we would have to provide a registry, right?#2021-06-1511:51wcalderipeSo, I assume Malli will check the value against every keyword type of a field, even when using an AND operator and the first check fails, in order to get a complete errors map in the first run :thinking_face:#2021-06-1514:31ikitommiyes, for the last one. See https://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D.#2021-06-1517:04wcalderipeGot it.. thanks @U055NJ5CC#2021-06-1519:20VladislavHi! I try to use malli as coercion&schemas with swagger ui. If i try serialise recursive schemas with registries, i get some "Could not resolve reference" of keys of recursive values. it is according to plan and swagger bad, or its just undone feature?) looks like there is some missing defs in swagger.json#2021-06-1519:43ikitommi@UNJE65T24 It might be that the references should copied to top-level swagger. If you can provide a minimal repro, would help solving#2021-06-1519:43ikitommialso, what version of swagger-ui are you using?#2021-06-1520:47Vladislav@U055NJ5CC i use swagger-ui bundled with
[metosin/reitit-swagger-ui "0.5.13"]
[metosin/ring-swagger-ui "3.36.0"]
guess it 3.XX#2021-06-1520:49Vladislavi'v got example of that malli#2021-06-1520:54Vladislavif i malli.json-schema/transform this (it is used in swagger gen, right?) i get something like#2021-06-1520:54Vladislavand it has no valuable
#/definitions/Expression
as you can see#2021-06-1607:00Vladislavshould i submit an issue to github?)#2021-06-1607:05ikitommiplease do#2021-06-1607:11Vladislavvoila https://github.com/metosin/malli/issues/464#2021-06-1608:03Yehonathan SharvitWhat's the rationale behind the fact that a set is considered as :sequential ?
For instance
(validate [:sequential string?] #{"aa"}) ;; false#2021-06-1608:19ikitommiit’s following the clojure way:
{:sequential (m/-collection-schema {:type :sequential, :pred sequential?})}
(sequential? #{}) ; => false#2021-06-1608:20ikitommiyou should be able create custom collections types easily that access both sequentials and sets.#2021-06-1608:50Yehonathan Sharvit@ikitommi Could you share a pointer to the documentation about custom collection types?#2021-06-1609:16ikitommisure: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L813. Documentation PRs most welcome 🙂#2021-06-1609:16ikitommiin use: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1925-L1927#2021-06-1610:06Yehonathan SharvitHow it would look like to create a custom :sequential-or-set predicate?
And where would i put its definition?#2021-06-1615:45ikitommi1. def a Var and use it instead of the type (keword), like with Reagent
2. add it to a registry, see README for alternatives#2021-06-1615:46ikitommi(def SequentialOrSet (m/-col...))
(m/validate [SequentialOrSet :int] [1])
#2021-06-1706:04Yehonathan SharvitOk. Thanls#2021-06-1611:23borkdude@viebel sequential means: it has a defined order. sets are not ordered, similar to maps, although you can create sequences out of them#2021-06-1614:08Yehonathan SharvitYeah. It makes sense. But still surprising.#2021-06-1614:08Yehonathan SharvitI think we need a name and a predicate for a bunch of things that could be either in a set or in a sequence#2021-06-1614:09Yehonathan SharvitMy use case is a function that receives a bunch of ids and return the entities whose id is contained in the bunch#2021-06-1617:53ikitommiyou could always say [:or [:sequential :int] [:set :int]].#2021-06-1617:53ikitommior, one could always transform the values always to sets/vectors.#2021-06-1618:05eskoscoll? works as well, if you don't mind using predicates directly#2021-06-1618:30ikitommi(coll? {}) ;=> true#2021-06-1618:33ikitommicoercion is always an option:
(m/decode
[:set :int]
[1 2 3]
(mt/collection-transformer))
; => #{1 2 3}#2021-06-1621:10deadghostAny recommendations on how to approach extracting out a registry from a schema? For example:
[:map
[::id int]
[:name string?]
[::country {:optional true} string?]]
to
{::id int?
::country string?}
and as a nice to have, the schema simplified to use the new registry:
[:map
::id
[:name string?]
[::country {:optional true}]]
Use cases:
• Simplify large schemas
• Finding differences in semantics
• Refactoring multiple schemas to use a shared registry#2021-06-1702:07escherizeHello, I want to build a general data-building ui, that takes a description of the datastructure as input (a malli schema, this time around). I couldn’t find much prior work in this regard. I have a half-baked prototype here: https://escherize.com/w/data-desk/
So far, that mostly works for int? string? boolean? :vector and :map and combinations of them.
So do you know of anything similar?#2021-06-1702:43escherizeAs for the implementation, I’m using a local atom and this multimethod: https://github.com/escherize/data-desk/blob/main/src/data_desk/views.cljs#L37#2021-06-1706:05Yehonathan SharvitI'd like to validate inside a function that the function is called with valid arguments.
Should I use :=> ? How exactly?#2021-06-1706:21ikitommiREADME should cover that#2021-06-1706:21ikitommihttps://github.com/metosin/malli#function-schemas#2021-06-1709:12Yehonathan SharvitThe readme doens't cover it#2021-06-1712:48Yehonathan SharvitHere is what I am doing#2021-06-1712:51Yehonathan Sharvit(def =>plus [:=> [:cat int? int?] int?])
(defn plus [x y]
(when-not
(m/validate (second =>plus) [x y])
(throw (ex-info "invalid input" {})))
(+ x y))#2021-06-1712:51Yehonathan SharvitIs there a more idiomatic way?#2021-06-1715:58ikitommithere will be malli.instrument, while waiting, there is malli-indtrument - https://github.com/setzer22/malli-instrument#2021-06-2007:12Yehonathan SharvitVery cool!
Any reason why it's not yet part of malli?#2021-06-2007:16ikitommijust time to merge & cleanup, the original issue is here: https://github.com/metosin/malli/issues/349#2021-06-2007:18ikitommithe function checker code shuould be reused between malli.generate and malli.instrument.#2021-06-2107:43Yehonathan SharvitIs someone already working on this consolidation?#2021-06-2108:10ikitommiplease ask on the issue. on my summer backlog if no-one has time. need this too after the vacations.#2021-06-2209:07Yehonathan Sharvitok. will do#2021-06-1719:55deadghostIs there something I can do to make the results of mu/union more condensed?
(mu/union (mu/union nil? empty?) empty?)
=> [:or [:or nil? empty?] empty?]
#2021-06-1804:53ikitommi@escherize looks good! haven’t seen anything library-quality impls of malli-forms. Have done one, but not complete and coupled to the use case.#2021-06-1804:55ikitommi@deadghost there should be an optional optimizer in malli. @miikka did a prototype of such some time ago: https://github.com/miikka/boolean-simplifier#2021-06-1804:57ikitommi[:or [:int {:min 10}] [:int {:max 5}] [:int {:min 0, :max 20}]] could be just :int.#2021-06-2006:49ikitommihttps://github.com/metosin/malli/blob/master/docs/tips.md#collecting-inlined-reference-definitions-from-schemas, ping @deadghost#2021-06-2117:52ikitommimalli.plantuml , the 23 loc monster 🙂#2021-06-2117:53ikitommihttps://github.com/metosin/malli/pull/468#2021-06-2118:00emccue> #(apply println %&)#2021-06-2118:00emccuewhat is that syntax#2021-06-2118:01emccue%ߥ-06-2118:01dvingohttps://clojure.org/guides/weird_characters#_n_anonymous_function_arguments#2021-06-2216:36ikitommi#(apply println %&) = println awesome#2021-06-2119:29alpoxOut of interest: Is there something in malli like a metaschema that can be used to validate a malli schema?#2021-06-2119:47Ben SlessMetacircular malli?#2021-06-2120:02alpoxMetacircular? 😄 You got me confused#2021-06-2120:18alpoxI just read up on Metacircular. I guess I am talking about something in that direction 😉
Basically a Malli schema that can be used to validate if my data at hand is a valid malli schema. If it was just a special keyword to refer to the metaschema while the metaschema is not directly written as a malli schema itself that would probably also be usable enough.#2021-06-2120:58ikitommiping @alpox#2021-06-2121:03alpox@ikitommi Oh, I really should have seen that in the history.. Thanks, that sounds promising!#2021-06-2209:08Yehonathan SharvitWhat's the recommended location for function schema definition?
Should it be on the same namespace as the function or in a separate namespace?#2021-06-2214:27ikitommiNo strong opinions in malli for that. My favourites:
1. Inlined (the plumatic syntax)
2. just before the functions (`m/=>` is as long as defn , looks good#2021-06-2214:27ikitommiwhat do you think?#2021-06-2214:45Yehonathan SharvitSomeone reviewed my code that added m/=> for each and every function in the core namespace of a lib (around 10 functions) and he thought it was polluting the namespace#2021-06-2214:46Yehonathan SharvitI tend to prefer to have the schemas near the code as schemas serve also as documentation#2021-06-2215:13eskosI prefer the Plumatic style as it's more about upfront (meta) data expression, m/=> feels unnecessarily clever and sort of indirect/IoC hellish since it comes after the function declaration. But that's just my two cents 🙂#2021-06-2314:38Lucy Wang+1 for this. Being able to place the schema right near to the argument is way more intuitive and maintainable than having them separate from each other.#2021-06-2215:14emccueOne option would be this#2021-06-2215:14emccuehttps://github.com/galdre/morphe#2021-06-2215:15emccuemake an aspect for it#2021-06-2215:16emccue^{::m/aspects [(contract [:=> [:cat int? int?] string?])]}
(m/defn some-fn [x y]
(str (+ x y))#2021-06-2215:22Yehonathan SharvitDoes malli support plumatic style? how?#2021-06-2216:26ikitommiwip, https://github.com/metosin/malli/pull/305#2021-06-2317:16respatializedI'm wondering whether function metadata maps may be suited to this purpose, the way spec does with :pre/:post validation inlined in an ordinary defn form, without needing a macro for function definitions. There may be a good reason to not overload the "canonical" keywords and instead use something like :malli/schema to inline it, but having it right in the form using metadata allows tests to be easily derived from a self-describing function with meta.
https://clojure.org/guides/spec
Was there something in your experience with spec-tools that led you to believe this isn't a good idea or the optimal solution for malli?#2021-06-2317:51ikitommithe plumatic syntax has been out since 2013 and I believe it's the most used syntax, by far. I have used it, it's just good.#2021-06-2317:53ikitommiI like also the`:malli/schema` suggestion, the malli.instrument could be configured to support multiple sources like:
1. the => registry
2. the :malli/schema metadata
3. plain old function schema inferrer#2021-06-2317:54ikitommiand, to be compliant:
4. the spec registry#2021-06-2216:29Yehonathan SharvitOk#2021-06-2216:31Yehonathan Sharvit@ikitommi In your previous message, you wrote:
> just before the functions (`m/=>` is as long as `defn` , looks good
What do you mean by m/=> before the function?
In malli's README, m/=> comes after the function definition#2021-06-2216:35ikitommithis works too:
(m/=> plus [:=> [:cat [:* :int]] :int])
(defn plus [& ns] (apply + ns))
(plus 1 2 3)
; => 6
#2021-06-2216:37Yehonathan SharvitNice. In fact malli treats function names as symbols#2021-06-2216:38Yehonathan SharvitSo it doesn't matter if you create the schema before the function is defined#2021-06-2314:38Lucy Wang+1 for this. Being able to place the schema right near to the argument is way more intuitive and maintainable than having them separate from each other.#2021-06-2317:16respatializedI'm wondering whether function metadata maps may be suited to this purpose, the way spec does with :pre/:post validation inlined in an ordinary defn form, without needing a macro for function definitions. There may be a good reason to not overload the "canonical" keywords and instead use something like :malli/schema to inline it, but having it right in the form using metadata allows tests to be easily derived from a self-describing function with meta.
https://clojure.org/guides/spec
Was there something in your experience with spec-tools that led you to believe this isn't a good idea or the optimal solution for malli?#2021-06-2321:31escherizeI’d like to attach error messages to specific parts of my schema. So far I have the following working, but I’m not sure if it’s the best implementation.
(def MySchema
[:map
[:something/one {:error/message "one"} int?]
[:something/two {:error/message "2"} int?]
[:something/three {:error/message "three"} int?]
[:something/four {:error/message "four"} int?]])
(validate MySchema {})
;; => #:something{:one ["one"], :two ["2"], :three ["three"], :four ["four"]}#2021-06-2321:33escherizeI am using the following to get this working:
(defn find-error-message [schema k]
(let [value (atom nil)]
(walk/postwalk
(fn [x]
(when (and x (vector? x) (= (first x) k) (map? (second x)))
(reset! value (second x)))
x)
schema)
(:error/message @value)))
;; then, in humanize:
(me/humanize ...
{:errors
(assoc me/default-errors
:malli.core/missing-key
{:error/fn (fn [{:keys [in]} _]
(or
(find-error-message schema (last in)) ;; is (last in) reliable here?
(str "Missing key: '"(last in)"'")))})})#2021-06-2321:38escherizeI’m not sure if using (last in) in the :error/fn is reliable though.#2021-06-2404:56ikitommi@escherize maybe:
(-> [:map
[:something/one {:error/message "one"} int?]
[:something/two {:error/message "2"} int?]
[:something/three {:error/message "three"} int?]
[:something/four {:error/message "four"} int?]]
(m/explain {})
(me/humanize {:resolve me/resolve-root-error}))
; => #:something{:one ["one"]:two ["2"], :three ["three"], :four ["four"]}#2021-06-2404:58ikitommithat’s in master, but not released. can be used, but will most likely change it to be a separate step, something like:
(-> Schema
(m/explain {})
(me/root-causes)
(me/humanize))
; => #:something{:one ["one"]:two ["2"], :three ["three"], :four ["four"]}#2021-06-2405:01ikitommianyway, it’s generic, so will find the top-most error defined:
(-> [:cat {:error/message "oh no!"}
[:? [:map-of :string :any]]
[:* :boolean]]
(m/explain 123)
(me/humanize {:resolve me/resolve-root-error}))
; => ["oh no!"]#2021-06-2405:02ikitommithere is a perf hit of recurring all errors towards root, but it can be later removed with an extra step of preparing a me/humanizer.#2021-06-2405:57escherizeAwesome. I’ll look into this. Thanks @ikitommi #2021-06-2501:55pinealanHi there! I’m trying to use malli as a parsing+coercion library for external data. I’m a bit lost on what the schemas should look like and what function should I be using.
An example is I want to transform a map with shortened key names like this {"a" "11", "b" "10.01"} into this {:ask 11.0 :bid 10.01}. Another case is sequential data, which may look like this ["11" "10.01"], and I would want to get the same data back.
I know it’s possible to do some of these steps without malli, i.e. rename-keys or spec/explain with :sequential, but it felt like there should be builtin idioms from the library/schema definitions that helps with the extra transformations that I am performing. Am I just missing something?#2021-06-2507:03ingesolI’m no expert, but I think you want encode/decode. See https://github.com/metosin/malli#value-transformation. You could write your own custom key-transformer that maintains a mapping of the abbreviations you want.#2021-06-2507:04ingesol(mt/key-transformer {:encode your-ns/kw->abbr
:decode your-ns/abbr->kw})
#2021-06-2507:07ingesolSchema for map would be
[:map-of :keyword :number]
Or whatever more specific number type you want.#2021-06-2507:07ingesol^^^ @achan961117#2021-06-2507:11ingesolSomething like this in the end (non-tested code)
(m/encode [:map-of :keyword :number]
{:ask 11.0 :bid 10.01}
(mt/transformer
(mt/string-transformer)
(mt/key-transformer {:encode your-ns/kw->abbr
:decode your-ns/abbr->kw})))
;; =>> {"a" "11", "b" "10.01"}#2021-06-2513:47Gary BergerHello when using map syntax is there a way of describing a oneOf relationship such that:
[:map
[:or
[:opt1 integer?]
[:opt2 integer?]]]
will allow either :opt1 or :opt2?#2021-06-2513:55emccuedo you intend that or to be inclusive or exclusive#2021-06-2513:57emccue[:and
[:or [:map [:a int?]]
[:map [:b int?]]]
[:map [:x int?] [:y int?]]]#2021-06-2517:35Gary BergerJust for clarity and not sure there is an easier way but exclusive or looks like
(m/validate
[:and
[:or
[:map [:p integer?]]
[:map [:q integer?]]]
[:not
[:and
[:map [:p integer?]]
[:map [:q integer?]]]]]
{:p 123 :q 123}) => false#2021-06-2513:58emccuei'm curious if there is a better answer that generates gooder, but this would do the right validation#2021-06-2516:15Gary Bergerexclusive thanks @emccue#2021-06-2517:09ikitommi@garyberger Usually, :and and :or are the way to go, see example in http://malli.io but there is also a declarative map-keys-relations poc: https://github.com/bsless/malli-keys-relations#2021-06-2821:44borkdudeIn case you didn't see it yet: https://github.com/piotr-yuxuan/malli-cli#2021-07-0115:36ikitommiThanks! Added link to README#2021-06-2822:47Gary BergerI will open an issue but this spec returns NPE with explain
(deftest xor-test
(is (= false
(specs/explain
[:and
[:or
[:map [:network integer?]]
[:map [:snapshot integer?]]]
[:not
[:and
[:map [:network integer?]]
[:map [:snapshot integer?]]]]] {}))))#2021-07-0115:36ikitommiAnswered, could not reproduce with malli 0.5.1#2021-07-0115:36ikitommiThanks! Added link to README#2021-06-3009:31Richard HundtHi everyone. I've got a malli schema of around 1500 sloc, with quite a lot of :multi schemas. I'm using it for transformation, and to speed up the runtime, I'm using malli/encoder and malli/decoder , however creating these transformers takes over a minute with version 0.5.1. I'm not sure how to reason about the complexity there, as it seems like I've got some combinatorial explosion (start-up time increases non-linearly in the size of the schema). Has anyone else run into this sort of thing? Any tips?
EDIT: I don't really see the need for speculative parsing or backtracking, since all the :multi schemas have discriminators which :dispatch can use to disambiguate, but there must be a lot of that going on, because plain calls to malli/decode without the "compiled" versions returned by malli/decoder take minutes themselves (not using regex schemas at all, either). 😕
(I'm running an Intel(R) Core(TM) i7-10875H)#2021-07-0115:39ikitommithat's not good @U026F3GM8NA. Could you create an issue out of this with a repro code showing the slowness. Thanks#2021-07-0118:03Richard HundtWill do.#2021-07-0206:29Ben Sless@U055NJ5CC could it be related to into transformer using satisfies? It's notoriously slow and GC intensive#2021-07-0207:16ikitommiit could, easy to see from flamegraphs.#2021-07-0207:40Ben SlessI can already guess from my experience with recursive calls to satisfies? , you'll see 95% of CPU is wasted on it#2021-07-0211:35Ben SlessIf -into-transformer was turned into a protocol, how would you make it work with cljs? fn? there is very different from in clj and won't be covered by a simple extend.
Possible to do in cljs what was done for Schemas in clj?#2021-07-0113:01pithylessIs there an existing PR or issue related to "dependent schemas" (i.e. returning a derivative schema, based on schema and value) that was mentioned on Jacek's podcast?#2021-07-0115:49ikitommiI don't think there is an issue yet.#2021-07-0115:51Ben SlessDo you feel like there might be a need for a getting started guide?
While the readme is comprehensive, I get the impression new users have a hard time building up beyond the simple examples. Something which starts from a simple base case, expands on it and illustrates malli's capabilities and features might be useful#2021-07-0116:07ikitommiThat would be awesome. Everything under docs gets published to cljdoc. Could also be a separate documentation site, if someone has time to work with that.#2021-07-0116:08ikitommiexamples of good doc sites (and tools to create them) would be a good start#2021-07-0116:28respatializedstay tuned on that one, I'm working on a heavily malli driven static website generator that uses itself to document itself#2021-07-0116:15Ben SlessDocs and cljdoc are good enough imo, but I'm more of a content over presentation kind of guy#2021-07-0116:18Ben SlessI'll start writing something. do you prefer adoc or md format?#2021-07-0116:18Ben Slessand on another note, any updates regarding the performance MRs I opened?#2021-07-0116:30ikitommiboth formats good, whatever you prefer. Adoc is better, right? The perf PRS (and all others), will check those soon. Just started my vacation, should have more time to invest in malli now.#2021-07-0116:55Ben SlessAren't you supposed to take time off on vacation? 🙂#2021-07-0116:56Ben Slesswho am I kidding if I had time off I don't think I'd be able to keep away from the computer for more than a day#2021-07-0117:14ikitommi6 weeks, should have time for both 😉#2021-07-0117:24Ben SlessLooks wonderful. Have fun!#2021-07-0117:51borkdude6 weeks, wow#2021-07-0123:12gregIs it possible to write mali, convert it somehow to spec and use it for spec/fdef and orchestration?#2021-07-0209:19ikitommino atm, but could be. Thing is that you can present things with malli that are impossible to present as specs, so it would not be complete like malli->JSON Schema , which is lossy. Still, I think 95% would work ok. There is malli-instrument library for doing the same with malli, if that’s ok for your use case.#2021-07-0214:09gregMakes sense. I wouldn't mind losing some of the malli features when instrumented as a spec.
Some of the libraries (like integrant's pre-init-spec) support only spec.
Although in my case, I use spec mostly for instrumentation of some bits. I was asking for such a conversion, as it sounds like the non-invasive way of trying malli in an existing project.
Anyway, I will take a look at malli-instrument. Thanks.#2021-07-0207:18rmcvHow do I provide sci options in malli decode? Trying to use java class with no luck.
(m/decode [:int {:decode/string '(fn [_] (.toEpochMilli (java.time.Instant/now)))}]
"whatever"
{:classes {'java.time.Instant java.time.Instant}}
mt/string-transformer)#2021-07-0209:21ikitommilooking at the code, it’s :malli.core/sci-options key in options, see https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1792-L1813#2021-07-0214:53rmcvtried this
(m/decode [:int {:decode/string '(fn [_] (.toEpochMilli (java.time.Instant/now)))}]
"whatever"
(merge
(m/-default-sci-options)
{:malli.core/sci-options {:classes {'java.time.Instant java.time.Instant}}})
mt/string-transformer)#2021-07-0214:53rmcvbut can’t see it in https://github.com/metosin/malli/blob/eea10aa8c668a16f88d9fc66fce0616a7f9137b4/src/malli/sci.cljc#L10#2021-07-0209:41ikitommi@ben.sless merged the loop unrolling perf PR, great for common cases and as it’s just clj, doesn’t make the cljs bundle size any bigger, big thanks!
;; 164ns -> 36ns
(let [valid? (m/validator [:and [:> 0] [:> 1] [:> 2] [:> 3] [:> 4]])]
(cc/quick-bench (valid? 5)))
#2021-07-0210:45ikitommimerged the other perf PR too:
;; 150ns -> 39ns
(let [valid? (m/validator [:map [:a :any] [:b :any] [:c :any] [:d :any] [:e :any]])
value {:a 1, :b 2, :c 3, :d 4, :e 5}]
(cc/with-progress-reporting
(cc/quick-bench (valid? value))))#2021-07-0210:54ikitommi#2021-07-0211:22Ben SlessNice 😁#2021-07-0509:47Richard Hundt@U055NJ5CC I've created https://github.com/metosin/malli/issues/472 with a test case which demonstrates the issue.#2021-07-0511:01Ben Sless@U026F3GM8NA I profiled you example, it looks like CPU mostly goes towards parsing the schema. If you wrap every def-ed schema with a schema constructor it seems to be way more efficient, would you mind trying?#2021-07-0511:02Richard HundtD'you mean just wrapping it in a function which returns the structure? (I'm not sure what you mean by "schema constructor")#2021-07-0511:06Ben Slesslike so;
(def A1
(m/schema
[:map
{:encode/test {:leave encode} :decode/test decode}
[:type [:= "A1"]]
[:value float?]
]))
#2021-07-0511:06Richard Hundtthanks, I'll try#2021-07-0511:06Ben Slessm/schema parses the children once#2021-07-0511:06Ben Slessyour implementation has no way to cache these results#2021-07-0511:07Ben SlessIt's a bother to do from scratch, you can take this#2021-07-0511:07Ben Sless#2021-07-0511:08Ben Slessthanks slack, very helpful#2021-07-0511:08Richard Hundtthank you#2021-07-0511:08Ben Slesshttps://gist.github.com/bsless/28a20f7e8f21f01fd93a788cf65444d4#2021-07-0511:09Ben Slessthere, more convenient#2021-07-0511:09Richard Hundtyeah, that's pretty much instant now, I'll try it on our production code#2021-07-0511:19Ben Sless👍#2021-07-0511:36Richard HundtPerfect. Thanks Ben! What shall I do with the github issue? Do I close it, or are you planning on making changes to parsing?#2021-07-0512:43ikitommiI closed the issue, no plans on making the parser faster or caching at the moment. Thanks for @ben.sless for digging into this 🙇#2021-07-0512:46Ben SlessI think it is worth to leave a comment on the issue and perhaps a note in the readme regarding references to uncompiled schemas#2021-07-0512:46Ben SlessAnd you're welcome 🙂
Turns out I was completely wrong regarding the transformers and satisfies?#2021-07-0512:47Ben SlessI see you left a comment with a link to the gist, so that would leave only the readme#2021-07-0513:14ikitommiif you have idea what and where to write this, please do 🙂#2021-07-0210:50ikitommi@ben.sless, great work. Saw the thread on Clojureverse about compiler optimizations. Would be great if things got automatically or magically faster. While waiting, writing performant code inside the libraries (on the hot perf path) is just being smart.#2021-07-0211:21Ben SlessThank you, I appreciate it.
Blame it all on the talk you gave on ClojureTRE 2019#2021-07-0313:15ikitommitoo hot, quick poke on instrumenting functions. moving functioin wrapping code from malli.generate into malli.core (just one fn) ->`m/-instrument`, can be used to enforce function schmas. First user will be malli.instrument utilities, but could be used directly in client code too:
(def pow2
(m/-instrument
{:schema [:=> [:cat :int] [:int {:max 6}]]}
(fn [x] (* x x))))
(pow2 2)
; => 4
(pow2 "2")
; =throws=> :malli.core/invalid-input {:input [:cat :int], :args ["2"], :schema [:=> [:cat :int] [:int {:max 6}]]}
(pow2 4)
; =throws=> :malli.core/invalid-output {:output [:int {:max 6}], :value 16, :args [4], :schema [:=> [:cat :int] [:int {:max 6}]]}
(pow2 4 2)
; =throws=> :malli.core/invalid-arity {:arity 2, :arities #{{:min 1, :max 1}}, :args [4 2], :input [:cat :int], :schema [:=> [:cat :int] [:int {:max 6}]]}#2021-07-0313:17ikitommilike everything else, can be configured to behave differently. e.g. just printing out the errors (or to tap ’em):
(def multi-arity-pow
(m/-instrument
{:schema [:function
[:=> [:cat :int] [:int {:max 6}]]
[:=> [:cat :int :int] [:int {:max 6}]]]
:wrap #{:input, :output}
:report (fn [error props]
(println "\n" error "\n")
(clojure.pprint/pprint props))}
(fn
([x] (* x x))
([x y] (* x y)))))
(multi-arity-pow 4)
;:malli.core/invalid-output
;
;{:output [:int {:max 6}],
; :value 16,
; :args [4],
; :schema [:=> [:cat :int] [:int {:max 6}]]}
; => 16
(multi-arity-pow 5 0.1)
;:malli.core/invalid-input
;
;{:input [:cat :int :int],
; :args [5 0.1],
; :schema [:=> [:cat :int :int] [:int {:max 6}]]}
;
;:malli.core/invalid-output
;
;{:output [:int {:max 6}],
; :value 0.5,
; :args [5 0.1],
; :schema [:=> [:cat :int :int] [:int {:max 6}]]}
;=> 0.5#2021-07-0313:25ikitommioh, and the generators use this too, generator for :=> looks dead simple now:
(defn -=>-gen [schema options]
(let [output-generator (generator (:output (m/-function-info schema)) options)]
(gen/return (m/-instrument {:schema schema} (fn [& _] (generate output-generator options))))))#2021-07-0314:04ikitommidunno if there is a way to emit clj-kondo annotations from schematized fns, which are not vars :thinking_face:#2021-07-0317:08respatializedSay I've got a :orn schema that I'm matching multiple values against:
[:orn [:map1 [:map {:closed true
:description "map1"}
[:a :string]]]
[:map2 [:map {:description "map2"}
[:a :string]
[:b {:optional true} :string]]]]
is there a way to see which of these subschemas were responsible for a given value matching the top-level form - something like m/explain but for values that do conform to the schema?#2021-07-0317:11ikitommi@afoltzm try m/parse#2021-07-0317:16respatialized@ikitommi thanks, I knew there was likely something I was missing. my earlier examples didn't use :orn - it seems like with an unnamed :or schema only the conforming value is returned by m/parse. is there a way to return the matching schema for unnamed :or subschemas? the use case I'm thinking of here is matching values against some collection of schemas that may not necessarily be known in advance, and so can't easily be named with :orn.#2021-07-0317:17respatializedit seems like I may be able to implement something in terms of m/walk if it's not built in?#2021-07-0317:21ikitommicurrrently, no. Could add an property to :or to hint that parse should return the index of the branch or an option to parse like :malli.core/parse-indexes true. Or, you can just create :or with number-indexed branches:
(m/parse
[:orn
[1 :int]
[2 :boolean]]
true)
; => [2 true]
#2021-07-0317:21respatializedah, gotcha. I think numbered branches is likely the easiest workaround for the time being.#2021-07-0317:23ikitommi(defn or->orn [s]
(m/into-schema :orn (m/properties s) (map-indexed vector (m/children s))))
(or->orn [:or :int :boolean])
; => [:orn [0 :int] [1 :boolean]]
#2021-07-0407:16ikitommiwip: malli.instrument (https://github.com/metosin/malli/pull/471)#2021-07-0507:14olyis there a way to have different map layouts, I wanted to do something like this
[:map [:coord [:or
[:map {:closed true}]
[:map {:closed true} [:x int? :y int?]]]]]
{:coord {}}
{:coord {:x 1 :y 1}}
I know I can make the whole map optional but what about allowing an empty map or a map with fixed keys, I tried a few things but could not find a way.#2021-07-0507:44alpox@oliver.marks I think this works if you use for the second :map:
[:map {:closed true} [:x int?] [:y int?]]#2021-07-0507:47olyyeah that works for the second map, just not sure how to make it work also for an empty map I tried putting optional in various places and tried :or and a few other things oh I might have missed some brackets above but its more an an illustration of one of the methods I tried#2021-07-0507:48oly[:map {:closed true :optional true} [:x int?] [:y int?]]
also tried this but that does not work, I know I could make the :x and :y keys optional but really I want an all or nothing method#2021-07-0507:49alpoxI tried it with your :or above and that worked (after adding the missing brackets)#2021-07-0507:52alpox#2021-07-0507:53olythis passes ? {:coord {}} when I run it through m/valid i get missing keys or I was yesterday when I was testing 😛#2021-07-0507:53olynice, I will re check what I have maybe there was a different error which I did not spot#2021-07-0507:56oly@alpox
[:map [:coord [:or
[:map {:closed true}]
[:map {:closed true}
[:x {:optional false} int?]
[:y {:optional false} int?]]]]]#2021-07-0507:56olywhat about that, I think because the key do not have the optional map the :or is irrelevant anyway#2021-07-0507:58olyalthough perhaps thats it, perhaps its optional thats messing me up looking at your screenshot again I can see it fails when one of the keys are missing#2021-07-0508:09oly@alpox thanks for your rubber ducking, not an issue with malli I had 2 schmeas with similar name and was updating the wrong one 😕#2021-07-0508:10alpoxOups 🙂 np#2021-07-0508:29Ben SlessI constantly get the feeling that I'm missing a schema to talk about map schemas in terms of key-value pairs#2021-07-0508:30Ben Slessan annoying example to implement is mutual exclusion, such as a map which can have x, y and (z or u)#2021-07-0508:30Ben Slesswhere z and u have different value types, too#2021-07-0512:34ikitommiHear hear. I think something like malli-keys-relations should be there. With qualified keywords, it’s less boilerplate (but, still has):
[:or
[:map ::x ::y ::z]
[:map ::x ::y ::u]]
#2021-07-0512:34ikitommispec-style would be:
[:map ::x ::y [:keys/or ::y ::u]]
#2021-07-0512:36ikitommimaybe:
[:and
[:map ::x ::y ::z ::u]
[:keys/or ::z ::u]]
#2021-07-0512:39ikitommi… and the mandatory JSON Schema way for this: https://json-schema.org/understanding-json-schema/reference/conditionals.html#2021-07-0512:49Ben SlessJSON Schema's BNF was one of the motivations for it#2021-07-0512:50Ben Sless> map which contains at least one ::x where ::x is [:or [:x int?] [:y float?]]#2021-07-0512:57Ben SlessIf we had it we could define JSON Schema with malli (translating the ebnf is straightforward), then define encoders and decoders to transform between the two#2021-07-0516:28ikitommiadded a video of the new function instrumention. three apis: advanced, normal users and for the lazy: https://github.com/metosin/malli/pull/471#2021-07-0516:28ikitommicomments welcome#2021-07-0603:40emccueIs there a good thing to use for "I make no assumptions about what this is"#2021-07-0603:41emccuelike any?, but for say something you can pass to next.jdbc#2021-07-0603:41emccueand won't be generatable#2021-07-0603:46emccuewriting a spec for what would count as a "db" has been confusing me, so i want a give up hatch for a second#2021-07-0609:37ikitommiso, :any is not enough? You could say [:any {:gen/elements ["whatever"]}] to generate what you like.#2021-07-0613:11emccueI guess it is with the empty gen elements.#2021-07-0612:15ikitommimiu/collect! with pluggable collector. anyone interested in doing fspec -> malli collector?#2021-07-0612:18ikitommithe source:
(defn -accept-default [v]
(let [{:keys [ns name malli/schema] :as meta} (meta v)]
(when schema (m/-register-function-schema! (-> ns str symbol) name (miu/-unlift-keys meta "malli")))))
(defn collect! [{:keys [ns accept] :or {accept -accept-default}}]
(doseq [[_ v] (ns-publics ns)] (accept v)))#2021-07-0612:46ikitommiIdea: an external tool that infers the schemas from vars and adds the needed schema meta into vars (rewrites the source, I recall there are tools for that already, edamame?).#2021-07-0615:10robert-stuttafordi use edamame on the edn that i give to malli, but only because i have the exceptions edamame gives me to tell the user where they typed something wrong when they malform edn#2021-07-0622:06borkdude> rewrites the source
This would be rewrite-clj#2021-07-0813:30robert-stuttaforduse that too 😄#2021-07-0615:08robert-stuttafordMalli is lovely.#2021-07-0619:06ikitommigenerated responses from function schema definitions:#2021-07-0622:40emccue@robert-stuttaford I'll add to that - the malli syntax is solid even without an impl. We don't use malli at work yet, but we have already started using its syntax for describing data in (comment ...) blocks.#2021-07-0813:26robert-stuttafordthat's pretty cool @emccue! it has the same property as hiccup - ultra consistent data optimised for human reader#2021-07-0910:12jeroenvandijkIf I want to validate if something is a function, it seems I can’t do (m/validate fn? (fn [])), but I can do (m/validate ifn? (fn [])) or (m/validate [:=> [:cat] any?] (fn [])) . Is there an easy way to look up what predicates are supported?#2021-07-0910:12jeroenvandijkFor now it seems the source is the quickest way? I’m looking into the source now to check it https://github.com/metosin/malli/blob/50c6cf6f6438401d6de2c46a9542f4c10da76a86/src/malli/core.cljc#L1750#2021-07-0910:36alpoxThere is the section https://github.com/metosin/malli#mallicorepredicate-schemas in the docs but it appears not to list ifn? :thinking_face:#2021-07-0910:42jeroenvandijkyeah I found this through the README https://github.com/metosin/malli#fn-schemas I’m guessing omission in the docs#2021-07-0916:46ikitommicould you @U0FT7SRLP update the docs?#2021-07-0916:48ikitommiwhen instrumentation is done, there could be a separate doc for function schemas. Lot's of different aspects (validation, generative testing, instrumentation, linting, ...) to cover.#2021-07-1015:42ikitommioh, fn? is missing, will add that#2021-07-0916:48ikitommiwhen instrumentation is done, there could be a separate doc for function schemas. Lot's of different aspects (validation, generative testing, instrumentation, linting, ...) to cover.#2021-07-1015:41gregHi, I get my feet wet in malli for the first time. I watched a couple of videos, I read some articules and scrolled the malli readme, and I;m still a bit confused about transformers.
I'm trying to create a bigdec? pred and transformator for it, so for given data structure I could coerce selected values from string and number to bigdec.
I looked for some examples on the internet. There are some examples of using pre-defined transformers but no examples of custom transformers. I mean, I haven't checked the code of Malli, I guess, that is the ultimate source of info.
Questions I'm trying to answer now:
• what are the guidelines for writing transformers?
• how to define custom predictor (in similar fashion to string?)?
• how to marry a custom predictor with a custom transformer (so when coercing a data structure, the transformer can detect properties to coerce for my custom predictor)
Any hints, links, would be very helpful. Thank you 🙇#2021-07-1020:54ikitommigood place to start could be this: https://github.com/metosin/malli#simple-schema#2021-07-1020:54ikitommihas a custom type, with a custom transformer, generator, json-schema mapping & humanized error dedinition#2021-07-1115:00gregI had to miss this part. Thank you#2021-07-1115:07gregI used to use Clojure Spec before, but I have never used Plumatic Schema before. I know it might sound ridiculously silly, but I haven't matched in my couscous mind the word 'schema' with a data shape definition. I mean, it makes total sense now, just I so get used to the word 'spec'. I don't know how I was watching this videos and reading these posts without that basic vocabulary understanding 😅#2021-07-1205:06datranWhat's the best way to update the properties of a map entry? I've got a schema like [:map [:a :string] [:b :string]], and I want to transform it into something like [:map [:a :string] [:b {:my-date :foo} :string]].
mu/update-properties only works for the :map-level, so I tried combining it with mu/update and giving a path into the map, but that also doesn't seem to be what I want. Is there a simpler way to achieve this that I'm overlooking?#2021-07-1315:42ikitommiYou can walk the children or wait for https://github.com/metosin/malli/pull/466 to be merged#2021-07-1315:45ikitommi(defn map-children [f schema]
(m/-into-schema (m/-parent schema) (m/-properties schema) (map f children) (m/-options schema)))
#2021-07-1315:46ikitommithat's the idiomatic way to create a copy of a schema with some changes#2021-07-1315:45ikitommi(defn map-children [f schema]
(m/-into-schema (m/-parent schema) (m/-properties schema) (map f children) (m/-options schema)))
#2021-07-1310:25gregIs it possible to define a default value for a map entry in terms of other map entry, e.g.
there are two keys in a map: primary and secondary. Secondary if missing is set based on primary one.#2021-07-1310:26gregSchema expressing it could look like this:
[:map
[:primary string?]
[:secondary {:default-fn '(fn [m] (:primary m))} string?]]
#2021-07-1311:02ikitommiout-of-the-box no, but you can implement this is user space, using a custom transformer. I recommend reading the source code of mt/default-value-transformer. My guess is that you should register the transformer for the map-level, peek the entrys and given there is :default-fn , call it with the map value. Should be streightforward. Could be a nice example into docs/tips.md...#2021-07-1311:14gregThank you!#2021-07-1617:22gregI had a look today in the default value transformer, and based on what you wrote I've made an example of a default value exercised by given function instead of default value (in the current impl). I modified the default-value-transformer to do that. But at the end I realised that I could handle this case:
[:map
[:primary string?]
[:secondary {:default-fn '(fn [m] (:primary m))} string?]]
but I wouldn't be able to handle this one:
[:map
[:primary string?]
[:nested
[:map
[:secondary {:default-fn '(fn [m] (:primary m))} string?]]
because the given fn receives only current map/submap not the root map? How would you tackle that kind of transformation?#2021-07-1617:34ikitommisimplest would be just to add a normal transformer into the top-map, which would use normal clojure to transform the whole nested map.#2021-07-1617:38ikitommiyou could make it declarative thou, but it's not simple:
1. add the transformer to the top-level
2. add the declarations to any children
3. use the :compile hook to 1 to get access to the full nested schema
4. within it, walk the schema and collect all the declarations and the paths, create a function to transform the child values
5. on transformation, the "compiled" transformation is applied at the top, knows already what to do, is fast#2021-07-1617:42ikitommi(actually, it is simple) 😉#2021-07-1618:23greg> simplest would be just to add a normal transformer into the top-map, which would use normal clojure to transform the whole nested map.
you mean, without Malli? I know, that would be the simplest, but I'm trying to understand malli transformers thoroughly and this problem looks like a good one to understand how Malli transformers works.
> actually, it is simple) 😉
I believe you it is simple, but at my stage of understanding malli transformers I'm reading each your response 10 times and it is not enough 🙂
I mean i understand briefly some of the bits already, based on the code and docs, but it it still not enough to understand some of the terminology you use.
TBH the first three steps are not really clear to me.
What do you mean by
"add the transformer to the top-level"
"add the declarations to any children"
"use the `:compile` hook to 1 to get access to the full nested schema" - I briefly know how the compilation works (preprocessing of the schema and passing discovered data within closured fn into interceptor), but I don't understand the phrase "hook to 1"
---
Also, within (transformer) I noticed that interceptors are created from transformers (from :decoders or :encoders) and default (from :default-decoder or :default-encoder). default-value-transformer uses both.
I mean, I look at the code and I don't understand what is it about. Looks like something obvious once you know it. Can you tell what is it about?#2021-07-2714:28gregI got back to the topic, trying to find all answers to my question.
@U055NJ5CC would you be so kind to read it and correct or complement if something is missing in the notes below. If the notes are correct, I could polish them a bit, maybe add examples, and add it to the Malli docs if you wish.
There is a hierarchy of hooks that Malli handle when transforming values (in this order of priority):
• from schema properties: :decode/<name> e.g. :decode/math - provides a transforming fn for a schema enclosing it (to enable, transformer specifying the given name need to be applied e.g. (mt/transformer {:name :math})
• from schema type properties: :decode/<name> e.g. :decode/math - provides a transforming fn for a schema type enclosing it (to enable same as above)
• from transformer definition: :decoders, :encoders - provides a map of (schema -> transforming fn), so for the every such schema in the given value's tree, applies the given fn
• from transformer definition: :default-decoder, :default-encoder - provides a last resort transforming fn, so if none of the above transforming fn hooks do not apply, it takes this one. This fn just takes the whole schema, so it might be called a top level hook.
Moreover:
• :enter/`:leave` refers to the interceptor stages, so we can provide separate transforming fn, they hold directly final functor modifying the the value. The map holding :enter/:leave can be provided by either compiling function or directly.
• compiling is the technique exercising values closure. To use it, instead of transforming function, a map {:compile <compiling-fn>} needs to be given. Compiling fun it is a fn taking schema and schema options, and returning a transforming fn. That way, processing of schema specifics can be done once at the time of constructing transformer.
#2021-07-1312:08ikitommiWIP: function schema guide. Wanted to push all aspects into single page, so that it’s easier to read what is possible and how the things stack up. Is that any good? something missing feature-wise? have time to polish things this week, comments most welcome: https://github.com/metosin/malli/blob/08111040566e2d39ecff62dfccdcb834f7a23140/docs/function-schemas.md#2021-07-1312:09ikitommiPR is here for comments & fixes: https://github.com/metosin/malli/pull/471#2021-07-1312:30gregSo actually orchestring project could be done in terms of m/=>, (mi/start!) and (mi/stop!). Am I correct?#2021-07-1312:39ikitommiyes.#2021-07-1312:21gregJust a note about orchestrating projects using Malli schemas.
I wanted to setup orchestration using malli in a way similar I used to do it with Orchestra. I checked out https://github.com/teknql/aave, https://github.com/setzer22/malli-instrument and https://github.com/CrypticButter/snoop.
Snoop: I've found it the best one for that purpose. Works fine with functions evaluated in REPL, readable error messages. A bit odd way of enabling it (by using JVM flag), on the other hand once set in dev and test aliases, it just works, for REPL and tests. Also 'https://github.com/CrypticButter/snoop#inside-the-prepost-map' syntax for adding function schemas is great.
malli-instrument: similar syntax to spec (call instrument-all to instrument), but there are problems with orchestration when you modify and evaluate continuously schemas and functions within REPL.
aave: I have had several looks at this library, and its README and TBH I have no idea how to use it. I guess I'm missing some relevant piece of information regarding instrumentation with Malli.#2021-07-2303:11Lucy WangI also find snoop very handy. To keep the schema inline with the arg name is a huge win
(>defn xy*2
[(x :int) (y :int)]
[=> :int]
(* 2 x y))#2021-07-2311:28gregAt first it was weird to me that Snoop supports so many options for providing inline schemas, but at the end I used 3 variants and at the 3rd time I found my flavour (just plain [:input :output]). So I'm kinda glad that Snoop doesn't enforce to use one particular syntax 🙂
I'm surprised that not that many people use orchestration. Seriously, it is one of the killer features of Clojure. And with Malli and its data-driven approach, it is so pleasant and concise to write and modify specs/schemas.
I'm aspiring to deliver to market an accounting software and when refactoring some of the code that does calculations or refactoring, the code orchestration is such a helper, always watching and keeping that invariants are valid.#2021-07-2404:15Lucy Wang@U023TQF5FM3 By "orchestration" do you mean https://github.com/jeaye/orchestra ?#2021-07-2404:18Lucy Wang> at the 3rd time I found my flavour (just plain `[:input :output]`).
Which flavor are you talking about?#2021-07-2416:47gregI prefer providing schemas in a two elements vector following the params vector.
(>defn add [x y]
[[:cat int? int?] int?]
...)
I found it the nicest to read 🙂
Here is an example:
https://github.com/CrypticButter/snoop#more-convenient-notations-that-work-when-using-defn#2021-07-2416:48greg> By "orchestration" do you mean https://github.com/jeaye/orchestra ?
I use the "orchestration" term for instrumentation of selected methods in a project, so you provide specs/preds/invariants and the tool using instrumentation watches the instrumented functions, either using Orchestra & spec or Snoop & Malli, within your REPL and tests.
I think the distinction between these two, is that instrumentation is a technique, while orchestration is a practice that leverages instrumentation.#2021-07-1312:36ikitommi@grzegorzrynkowski_clo did you check the guide I just posted? There will be a malli.instrument ns, with common utilities for spec/orchestra-style usage. It's all built on existing function (var) registry, so no new macros/defn-syntax. It would be great if all defn-wrapping libs could use malli.instrument internally, would keep thing more coherent. Not sure if that is possible thou. All existing libs (aave, Snoop, malli.instrument) are great, just that the core instrumentation belong to the core. Better defn wrappers more welcomed outside.#2021-07-1312:45gregI've seen it, just was looking for some ready to go solution right now. When malli.instrument ns comes, most probably I will reevaluate my setup 🙂#2021-07-1313:19gregIs there a function to validate schemas' data structures? I was playing with Malli and transformers and I was trying to figure out what is wrong with my custom schema. I tired to coerce data using string-transformer and it didn't work. It turned out the schema of my schema was wrong.
Here is an example showing my problem:
(def Order
[:map
[:qty number?]
[:price string? float?]])
(comment
(m/validate Order {:qty "1" :price "1.1"}) ; => false
(m/validate Order {:qty 1 :price "1.1"}) ; => true
(m/decode Order {:qty "1" :price "1.1"} mt/string-transformer) ; => {:qty 1.0, :price "1.1"}, but I expected {:qty 1.0, :price 1.1}
(->> (m/decode Order {:qty "1" :price "1.1"} mt/string-transformer)
(m/validate Order))) ; => true
Schema [:price string? float?] is wrong. Unintentionally I forgot to remove string? . Still it is fairly small example, but within complex examples might be difficult to spot it. Would be nice to have a method checking schemas data structures. Is there something like that?#2021-07-1313:44emccueSounds like you want a schema for your schemas#2021-07-1313:57ikitommisomeone needs to define those for all schemas. Malli supports them already#2021-07-1617:22gregI had a look today in the default value transformer, and based on what you wrote I've made an example of a default value exercised by given function instead of default value (in the current impl). I modified the default-value-transformer to do that. But at the end I realised that I could handle this case:
[:map
[:primary string?]
[:secondary {:default-fn '(fn [m] (:primary m))} string?]]
but I wouldn't be able to handle this one:
[:map
[:primary string?]
[:nested
[:map
[:secondary {:default-fn '(fn [m] (:primary m))} string?]]
because the given fn receives only current map/submap not the root map? How would you tackle that kind of transformation?#2021-07-2303:11Lucy WangI also find snoop very handy. To keep the schema inline with the arg name is a huge win
(>defn xy*2
[(x :int) (y :int)]
[=> :int]
(* 2 x y))#2021-07-1416:35wcalderipeHey folks, I'm trying to use Malli to parse keys and values of a map to something else but I'm having a bit of difficulty understanding how and what function to use. I've tried m/decode and m/parse so far but not with much success.
How can I achieve a transformation like the one below using Malli?
;; input
{:foobar "baz"
:qux "999"}
;; output
{:foo :baz ;; Changes the key from :foobar to :foo
:qux 999 ;; Coerces the value from string to integer
}
For context, the use case I have in mind is parsing the response of an API to something else. I'm sharing it because maybe there's a better way to use Malli for schema checking of services we don't control plus parsing the data to something else.#2021-07-1417:07ikitommi@wcalderipe maybe:
(def decode
(m/decoder
[:map
[:foo :keyword]
[:qux :int]]
(mt/transformer
(mt/key-transformer {:decode #(get {:foobar :foo} % %)})
(mt/string-transformer))))
(decode
{:foobar "baz"
:qux "999"})
; => {:foo :baz, :qux 999}#2021-07-1417:20wcalderipeGot it.. thanks for taking the time to reply 🙌
I have another question between these lines. Can I reuse the same schema for validating the API response and do the whole transformation?
Let's say we're integrating a Todo API and we need to do some transformation on their response.
;; Todo API schema as it is
(def todo-api
[:map
[:name [:string {:min 1}]]
[:done [:boolean]]
[:create_at [:string]]])
(malli/validate todo-api {:name "Ask for help at #malli channel"
:done true
:create_at "1615007456"})
;; => true
1. Validate if responses are following the schema
2. Parse the response to something else as you've shown
;; Output
{:name "Ask for help at #malli channel"
:done true
:timestamp 1615007456} ;; Rename the key and transform the value
Do I need two schemas 1 for the input and 1 for the output in a scenario like this? :thinking_face:#2021-07-1510:14gregMy understanding is that Malli encourage to transform first and validate next.
For example, let's say data you are receiving are invalid. You decode, validate it. It turned out something is wrong you might want to return an original piece of data by transforming it back (using m/encode) and adding some info what's wrong.
If the response has a different schema by its identity, make sense to have a different schema.
Disclaimer, I'm not an expert in Malli 🙂#2021-07-1510:24gregSo should you have one schema or two, is question are these data same thing or two different things. And answer is not always clear. E.g. In layered architectures, like MVP, depends on implementation you might have several schemas (representation) of the same data.#2021-07-1512:18ikitommiif the web-facing values can be mapped automatically to internal values, you should be able to do the same transformation for the schemas too, e.g. define just one and generate the other. Reason for wanting to have two schemas could be apidocs: they should be generated from “web-facing” schemas, not the internal model.#2021-07-1512:18ikitommirelated: https://www.metosin.fi/blog/malli/#schema-transformation#2021-07-1512:20ikitomminot sure if there are all the needed helpers in malli.util , but easy to add if something is missing.#2021-07-1606:00wcalderipeThanks for taking the time @UBVL1LR5F
> E.g. In layered architectures, like MVP, depends on implementation you might have several schemas (representation) of the same data.
I think the problem I have in mind, the data means the same thing but it has a different shape inside the boundaries of my app. Malli will be used near to the edges of the app into something like an anti-corruption layer to translate an external concept to an internal one. That's renaming some keys and coerce some of the values too.#2021-07-1613:22gregIf the Malli resides only on the one side of the ACL architecture, on the internal one, I think you could work with one schema. If both would use Malli, then I would go with two.
Going back to you question
> Can I reuse the same schema for validating the API response and do the whole transformation?
I think yes, you could use one. You received response from external (behind ACL) system, you transform (including the keys) and then validate.#2021-07-1613:24gregIf you want validate first for some reason, you might need two.#2021-07-1708:24wcalderipeThanks for sharing @U055NJ5CC
My doubt is more a question of design than which functions to use.
@UBVL1LR5F got it...
> If you want validate first for some reason, you might need two.
Yeah, I think I'll end up with two instead of one because I was thinking of doing validations as well in development mode.
Thanks a bunch for the help!#2021-07-1512:45ikitommimerged malli.instrument PR into master. will be released as part of 0.6.0 later. test reports, feedback, improvements welcome! the new function guide is here: https://github.com/metosin/malli/blob/master/docs/function-schemas.md 😎#2021-07-1609:46hansbuggeHi, I'm experiencing something funny with humanize in Malli 0.5.1. When the wrapper function returns a string, then the :malli/error key gets an entry for each error, but when it returns anything but a string then there will be only one error under :malli/error.
(-> (m/explain [:and
[:fn (constantly false)]
[:fn (constantly false)]]
{:a :map})
(me/humanize {:wrap (constantly "a string")}))
;; => #:malli{:error ["a string" "a string"]}
(-> (m/explain [:and
[:fn (constantly false)]
[:fn (constantly false)]]
{:a :map})
(me/humanize {:wrap (constantly :not-a-string)}))
;; => #:malli{:error [:not-a-string]}#2021-07-1609:48hansbuggeAnd I'm having a hard time understanding why from looking at the source. Am I right that this is a bug?#2021-07-1609:56hansbuggeIt seems to be the form (if (-just-error? v) (into (vec e) v) v) in malli.error/-put, where -just-error? only returns true if the value is exactly a vector with one string#2021-07-1610:01hansbuggeMy problem is that we're using explainers and humanize to generate structured form validation output, where the wrapper passed to humanize generates a map with the error message and a couple of other fields. But this means we get at most one error in :malli/error.#2021-07-1613:48ikitommijust wrote a issue based on old PR to make the :and error handling more robust, e.g. take the first error form and accumulate to that. Not sure if that would help here, but coming anyway: https://github.com/metosin/malli/issues/476#2021-07-1615:45ikitomminice to see how things click while developing malli.
• in spec1, s/and flows conformed values, which is IMO not intutive, but needed for special cases (e.g. parsing sequential input), source of much confusion
• in spec1, s/cat forced things to be named, need you it or not
• in spec2, there might be a non-conforming variant and', most likely no cat' thou.
with malli, the simple syntax is the default and you can detail it (named branches, parsed childs) if needed. Spec has amazing ideas and personally is a big source of inspiration. The Malli evolution:
Given:
(defn distance [min max]
(- max min)
Simplest description:
;; int int -> int
[:=> [:cat :int :int] :int]
Adding names:
;; min:int max:int -> int
[:=> [:catn [:min :int] [:max :int]] :int]
Adding constraints on input (with new parsed utility schema):
;; min:int max:int [min < max] -> int
[:=>
[:and
[:catn
[:min :int]
[:max :int]]
;; here we want to used parsed values instead of orginal sequence
[:parsed
[:fn
{:error/message "min should be lass than max"}
(fn [{:keys [min max]}] (< min max))]]]
:int]#2021-07-1617:39gregHaha, I've spend a couple of hours trying to understand the logic behind it recursive traversal of transformers until I discovered it is at the Schema implementation, in the transformer- fn 😄#2021-07-1705:07mike_ananev@ikitommi , hi! I've tried to run static type checking using FileWatchers + clj-kondo. Can't see any effect for plus1. clj-kondo works well. malli version 0.6.0-snapshot#2021-07-1709:06ikitommidid you add the clj-kondo configs for malli? Not sure if that could be automatic in the future? Maybe @U04V15CAJ knows? https://github.com/metosin/malli#clj-kondo#2021-07-1709:11ikitommie.g.
✗ cat .clj-kondo/config.edn
{:config-paths ["configs/malli"]}#2021-07-1709:42borkdudeconfigs are never opted into automatically#2021-07-1709:43borkdudeby design, you should opt into those manually, for safety and consistency reasons#2021-07-1710:07ikitommiSounds right. But is there a default directory I could write the config file to, without breaking anything?#2021-07-1710:08borkdudeit's best to write into a directory with a fully qualified name like com.metosin.malli or so#2021-07-1710:08borkdudeto avoid conflicts#2021-07-1710:08borkdudeor just the domain name of your library#2021-07-1710:16ikitommigood idea. but despite being fully qualified, users need to enable that config in their projects, right? no option to "trust 'em all"?#2021-07-1710:17borkdude> by design, you should opt into those manually, for safety and consistency reasons#2021-07-1710:17borkdude;)#2021-07-1710:18borkdudeyou should just add that dir to your config paths, that's it#2021-07-1710:18borkdudeany tool can dump any config into the dir, but it's not always something a user wants to opt into#2021-07-1710:19mike_ananev@ikitommi you was right! I added {:config-paths ["configs/malli"]} to .clj-kondo/config.edn and now clj-kondo works well with malli static type checking#2021-07-1710:19borkdudeyesterday I had such a report where clojure-lsp dumped a config into the dir, but a user did not want to have this config#2021-07-1705:07mike_ananev#2021-07-1810:24Vincent CantinI have a schema that parses a value correctly but then the unparse operation returns :malli.core/invalid. Is it a known problem for some kind of schema or should I post a bug report?#2021-07-1811:39Vincent CantinTo reproduce:
(let [my-schema (m/schema
[:and
list?
[:catn
[:wrapper [:= 'value]]
[:wrapped int?]]])]
(->> (m/parse my-schema '(value 5))
(m/unparse my-schema)))#2021-07-1811:40Vincent Cantinm/unparse works if I remove list? from the schema, but then it's not the schema I need.#2021-07-1811:54Vincent CantinIs there an option in catn to specify the list of container of the sequence?#2021-07-1812:57ikitommicurrently no, but definetely it should.#2021-07-1812:57ikitommimaybe something like?
[:cat {:type :vector} :int :int]#2021-07-1812:59Vincent Cantinyes, but the name "type" is not descriptive enough as it relates to the container's type and not the sequence itself.#2021-07-1813:00Vincent CantinIn minimallist, I named it "coll-type" https://github.com/green-coder/minimallist/blob/all-work-and-no-play/src/minimallist/helper.cljc#L72-L76#2021-07-1813:02Vincent CantinRight now, I am trying to port my model from minimallist to malli in the Vrac project, so I have less things to maintain and it's better for the users if they face a mainstream library like Malli.#2021-07-1813:14ikitommihave you checked the bundle size? malli starts from ~2kb (just the minimal working set of schemas), but is quite big if all schemas are used (40kb?). with all the bells & whistles, it can be >100kb.#2021-07-1813:15ikitommiI’m not sure how :type and :coll-type are that different, but something like that would be good.#2021-07-1813:17Vincent Cantin:type will be fine#2021-07-1813:28Vincent Cantinthe bundle size won't be a problem, I can parse things offline.#2021-07-1818:16Vincent Cantin:tuple would also need to have the {:type :list} option.#2021-07-1810:30Vincent CantinWhile chaining parse and unparse, I realized that the API would be more friendly to -> if the parameter order of ?schema and value were exchanged.#2021-07-1813:02ikitommidid a quick run on improving performance of collection transformations, got easy x5 for CLJ in simple test:
(let [schema [:map
[:id :string]
[:type :keyword]
[:address
[:map
[:street :string]
[:lonlat [:tuple :double :double]]]]]
decode (m/decoder schema (mt/json-transformer))
json {:id "pulla"
:type "herkku"
:address {:street "hämeenkatu 14"
:lonlat [61 23.7644223]}}]
;; 920ns => 160ns
(cc/quick-bench
(decode json)))#2021-07-1813:05ikitommialso, coercion (transform + validate) is 5x faster with that data compared to Plumatic Schema. Which is kinda nice as have considered (the awesome) Plumatic as a performance reference.#2021-07-1813:05ikitommimerged in master.#2021-07-1813:05ikitommihttps://github.com/metosin/malli/pull/478#2021-07-1813:14Ben SlessI think you can close over the iteration completely#2021-07-1813:42Ben Sless@U055NJ5CC you can gain at least 10% and probably gain some mechanical sympathy by closing over the transforms:
(map
(fn [[k v]]
(fn [^Associative x]
(if-let [xe (.entryAt x k)]
(.assoc x k (v (.val xe)))
x)))
ts)
then reduce over them with -comp#2021-07-1813:47Ben SlessI can PR this if you'd like#2021-07-1818:58ikitommilooks good. But, I thing the varargs version of -comp is actually slow as it uses the sequence abstraction, could be rewritten to use iterator? PR (and perf test before & after) most welcome.#2021-07-1819:01Ben SlessI unrolled it to 16 args#2021-07-1819:21ikitommi👍#2021-07-1819:22ikitommialready this is much faster that the current:
(defn -comp
([] identity)
([f] f)
([f g] (fn [x] (f (g x))))
([f g h] (fn [x] (f (g (h x)))))
([f1 f2 f3 & fs]
(let [fs (into [f1 f2 f3] fs)]
(fn [x] (let [i (.iterator ^Iterable fs)]
(loop [x x] (if (.hasNext i) (recur ((.next i) x)) x)))))))#2021-07-1819:45Ben SlessI'll send an organized PR tomorrow, today is getting late#2021-07-1907:41Ben SlessI wonder if there's an optimal maximum arity#2021-07-1813:05Ben Sless👀#2021-07-1813:07ikitommiMy initial thought was that having transform and validation as separate steps can actually make it faster - as the created transformation chain is usually small enough to fit into the JVM inlining budget - while validation is always “complete” and generated more code. Having those in one sweep means more code. Haven’t looked at the perf profile, so just quessing.#2021-07-1813:10Ben SlessI was just thinking about this recently - would interleaving transform and validation be faster? Since it involves lots of iteration and allocation, we can't necessarily assume JIT friendly code would be faster than a single pass over two passes#2021-07-1813:08Ben Sless> entryAt
😄#2021-07-1813:09ikitommi#2021-07-1813:09ikitommithe initial code was using reduce for all, which does a lot of things.#2021-07-1813:10ikitommistack size from 36 -> 6 (on malli side).#2021-07-1813:11ikitommibut thanks @ben.sless for the validation perf, mostly same optimizations for transformers 🙇#2021-07-1813:12Ben SlessI thought it looked familiar :thinking_face:#2021-07-1813:12Ben Sless😄#2021-07-1813:17ikitommireduce-kv should be fast, but for some reason, it’s not always used instead of reduce. Recall there was a related bug in clojure for this. writing by hand is always fast 🙂#2021-07-1813:27Ben SlessHave you had a chance to look at my proposal on https://github.com/metosin/malli/issues/474?#2021-07-1813:46Ben SlessI have no idea how to compile my proposal to something efficient, though I assume it is possible and someone clever like nilern could do it#2021-07-1820:07Vincent CantinIt is possible in Malli to write a schema which represent Clojure's destructurations of a map? For example, something which would parse this kind of data:
{a :a
b :b
:keys [c d]
:e/keys [f g]
:as h}#2021-07-1907:49ikitommiI don't think there is. Does spec have something for this?#2021-07-1907:49ikitommiand what would be the expected parse result?#2021-07-1907:51Vincent CantinI don't know if spec can do something like that. In minimallist, I approach this problem by having map-of taking a schema of a pair [key value], so I am able to use :alt on the pair to valid multiple different cases where the key and the value are correlated.#2021-07-1907:52ikitommimaybe m/parse could have a custom user-defined property to allow users to give the parse & unparse functions for the given schema.#2021-07-1907:53Vincent CantinAn escape hatch, yes it would work, but the hardship would be on the user.#2021-07-1907:53ikitommicould you write a minimallist sample for that case? Sounds interesting#2021-07-1907:53Vincent CantinHere it is: https://github.com/green-coder/vrac/blob/diy-furry/src/vrac/model.cljc#L25-L30#2021-07-1907:56Vincent CantinThe escape hatch may not work well for the user in the case the model is using recursion, like in this example. That would become Malli -> fn -> Malli -> fn ...#2021-07-1820:15respatializedHi all! I've been working for a while on fabricate, a static website generator that takes advantage of malli in order to both provide a model for semantically valid HTML/Hiccup forms and to define its own order of operations. I wrote a post about my experience using malli schemas to define a finite state machine that dispatches functions on the basis of the schemas matched by the data passed in to them, which I have taken to calling "finite schema machines."
https://fabricate-site.github.io/fabricate/finite-schema-machines.html
I'd be really interested in what other users of malli think about this concept. It's highly experimental, but I've already found it quite interesting and useful.#2021-07-1909:30Ben SlessThis looks very interesting and I've had FSMs on my mind recently. Will need to give a deeper look#2021-07-1910:04Ben SlessWonder how this relates to dependent types#2021-07-1912:50respatializedI've never worked with dependent types, but my impression of malli is that you get a lot of the benefits of dependent types without a static type system. that said, when I think about how to validate/check these FSM definitions for cycles or non-termination at definition/compile time, I wonder how close I'm actually getting to reinventing a type system.#2021-07-1907:47ikitommi@afoltzm looks awesome!#2021-07-1920:20mike_ananev@ikitommi, do you have any plans to release malli 0.6.0 soon? Malli 0.6.0-snapshot has very exciting functionality about function schemas and instrumentation.#2021-07-2009:13ikitommino plans, but for the instrumentation part, the latest immutable SNAPSHOT version should be stable'ish (or depend on latest commit via deps)#2021-07-2009:13ikitommino plans, but for the instrumentation part, the latest immutable SNAPSHOT version should be stable'ish (or depend on latest commit via deps)#2021-07-2003:03escherizeIs there a malli schema for the clojure.core/defn form?#2021-07-2004:50mike_ananev@escherize https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defn-schemas-via-metadata#2021-07-2005:45escherizethanks for your response @mike1452. What I am looking for is a malli schema that will say “that clojure.core/defn form you have is good and not malformed” or not.#2021-07-2005:47escherizeI think the schema should match these, but there may be more:
['(defn f [a] 1)
'(defn g [a] (+ a a))
'(defn ^:ok g "hi" [a] (+ a a))
'(defn ^:ok g "hi" ([a] (+ a a)))
'(defn ^:ok g "hi" ([a] 1) ([a b] 2))]
#2021-07-2006:00escherizeI thought I was getting close, but can’t quite get it for all cases. Maybe someone here can figure it out
(def DefnSchema
[:cat
[:enum 'defn '-defn]
symbol?
[:? string?]
[:alt
[:cat [:vector symbol?] [:* any?]]
[:* [:sequential [:cat [:vector symbol?] [:* any?]]]]]])
(mapv #(m/explain DefnSchema %)
['(defn f [a] 1) ;;<- good
'(defn g [b] (+ b b)) ;;<- good
'(defn ^:ok g "hi" [c] (+ c c)) ;;<- good
'(defn ^:ok g "hi" ([d] (+ d d))) ;;<- fail
'(defn ^:ok g "hi" ([e] 1) ([e e] 2))]);;<- fail#2021-07-2011:10ikitommim/explain might hint why it's not right#2021-07-2011:10ikitommioh, you had that.#2021-07-2011:11ikitommiany hint about why they failed? (not near computer myself)#2021-07-2013:47danielnealIs there already a transformer that will work with query parameters - specifically to convert single values into vectors where the schema specifies a vector.
i.e.
(malli.core/decode [:map
[:a [:vector int?]]]
{:a "1"}
some-transformer) => {:a [1]} (as opposed to {:a "1"})
(malli.core/decode [:map
[:a [:vector int?]]]
{:a ["1" "2"]}
some-transformer) => {:a [1 2]}
This seems to work - but is it right?
(defn collection-transformer []
(malli.transform/transformer
{:decoders
{:vector
{:compile (fn [schema _]
(fn [x]
(if (vector? x) x [x])))}}}))#2021-07-2013:52ikitommi@escherize it should be:
(def DefnSchema
[:cat
[:enum 'defn '-defn]
symbol?
[:? string?]
[:alt
[:cat [:vector symbol?] [:* any?]]
[:+ [:schema [:cat [:vector symbol?] [:* any?]]]]]])#2021-07-2013:53ikitommie.g. not sequence :+ of sequence :sequence of sequences :cat, just sequence (`:+`) of sequences :cat.#2021-07-2013:58ikitommi@danieleneal maybe something like:
(m/decode
[:map
[:a [:vector int?]]]
{:a "1"}
(mt/transformer
(mt/transformer
{:decoders {:vector (fn [x] (if (string? x) [x] x))}})
(mt/string-transformer)))
; => {:a [1]}#2021-07-2014:06danielnealah cool, thanks, looks like I’m on the right lines 🙂#2021-07-2014:09ikitommiyou can use :compile if you want to access the schema ahead of time, like reading the separator per schma:
(def decode
(m/decoder
[:map
[:a [:vector {:separator ";"} int?]]]
(mt/transformer
(mt/transformer
{:decoders
{:vector
{:compile (fn [schema _]
(let [separator (-> schema m/properties :separator (or ","))]
(fn [x]
(cond
(not (string? x)) x
(str/includes? x separator) (into [] (.split ^String x ^String separator))
:else [x]))))}}})
(mt/string-transformer))))
(decode {:a "2"})
; => {:a [2]}
(decode {:a "1;2"})
; => {:a [1 2]}#2021-07-2014:25ikitommi@danieleneal https://github.com/metosin/malli/blob/master/docs/tips.md#decoding-collections#2021-07-2014:35danielnealnice!! Thanks :))))#2021-07-2014:29kennyI'm curious if Malli has considered allowing the default registry to be set in a less invasive way (i.e., not through jvm props)? The requirement to set a jvm prop touches many places in a large application and, I imagine, will cause future developer confusion by requiring every repl to be launched with that prop.#2021-07-2014:43ikitommitotally agree. the current way to enable custom registry is too much work and one can always define an immutable registry for the cases (multi-tenant env) when it matters. Just not sure what would be a best possible compromise between simple & easy here. Imperative programming with global state is not good either. Ideas?#2021-07-2014:44ikitommithere was a discussion somewhere some time ago..#2021-07-2014:46ikitommi1. immutable by default, swapping needs a custom jvm/compiler option
2. mutable registry by default, spec-like
3. immutable by default, but mr/set-default-registry! available without jvm options
4. something else#2021-07-2015:10kennyWe are very early in our usage of Malli, coming from a large use of Spec, so take anything I say with that grain of salt. I quite like the default mutable registry. Over the years, we have built up a large library of domain specs that are used all over the place. It's handy to be able to simply reference these specs by their keyword name. Of course, Malli may encourage different conventions (e.g., just write functions returning custom schema). I started down the path of creating our own immutable registry, but started to feel pain when I needed to pass my registry to every single Malli api call.#2021-07-2015:25mike_ananev@ikitommi I found breaking changes between 0.5.1 and 0.6.0-snapshot in a function m/validate
In 0.5.1, function m/validate returns true if data corresponds to spec. In 0.6.0-snapshot, function m/validate returns data, not boolean value true.
In 0.5.1, If data is not corresponds to spec, then function m/validate returns false. In 0.6.0-snapshot, function m/validate returns in some cases value false, in some cases returns value nil (for strings for example).#2021-07-2107:21ikitommioh, that’s not good @mike1452. These might be related:
• https://github.com/metosin/malli/commit/ae12531aede1ad936af7d7bde80543572cc907ae
• https://github.com/metosin/malli/pull/479
… could you retest with the new SNAPSHOT (`metosin/malli-0.6.0-20210721.071739-2`) and if the problem persists, please write an issue.#2021-07-2108:26mike_ananev@ikitommi thank you for quick reply! I'll check my tests and return soon.
You can use malli spec to spec your functions to catch errors, while you are developing malli.spec. 😂#2021-07-2110:11mike_ananev@ikitommi I found new strange behaviour during this cycle:
edn1 -> encode-value -> json -> decode-value -> edn2
In v 0.5.1, edn1 = edn2. All uuids and zoned-date-time encoded/decoded correctly, they are data of the same type.
In v 0.6.0-20210721.071739-2, edn1 != edn2
broken data for uuid and zoned-date-time - they are strings after decoding.
Can't reproduce this error on a simple schema. I've more complicated schemas, but cannot publish them here.
But in my code I do something like that.#2021-07-2110:12mike_ananev(do (def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
(def my-spec1 [:map
[:a :zoned-date-time]
[:b :string]
[:c :uuid]
[:d :int]])
(def my-spec [:map [:s1 my-spec1] [:s2 my-spec1]])
(def edn-value (mg/generate my-spec))
(def encoded-value (m/encode my-spec edn-value custom-json-transformer))
(def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
(def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
(def edn-value2 (m/decode my-spec encoded-value2 custom-json-transformer))
(is (m/validate my-spec edn-value2)))
#2021-07-2112:26Ben SlessIs this example sufficient for reproduction or is it just illustrative?#2021-07-2113:03Ben SlessSome schemas are missing from the example, too#2021-07-2113:27mike_ananev@UK0810AQ2 it is illustrative#2021-07-2116:36Ben SlessI MANAGED TO RECREATE IT
(do
(def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
(def my-spec1 [:map
[:a/c1 {:optional true} :uuid]
[:a/c2 :uuid]
[:a/c3 :uuid]
[:a/c4 :uuid]
[:a/c5 :uuid]
[:a/c6 :uuid]
[:a/c7 :uuid]
[:a/c8 :uuid]
[:a/c9 :uuid]
[:a/c10 :uuid]
[:a/c11 :uuid]
[:a/c12 :uuid]
[:a/c13 :uuid]
[:a/c14 :uuid]
[:a/c15 :uuid]
[:a/c16 :uuid]
[:a/c17 :uuid]
[:a/c18 :uuid]
[:a/c19 :uuid]
[:a/c20 :uuid]
[:a/c21 :uuid]
[:a/c22 :uuid]
[:d :int]])
(def my-spec [:map [:x/y [:map [:b/foo [:map [:a/s1 my-spec1]]]]]])
(def edn-value (mg/generate my-spec))
(def encoded-value (m/encode my-spec edn-value custom-json-transformer))
(def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
(def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
(def edn-value2 (m/decode my-spec encoded-value2 custom-json-transformer))
(m/validate my-spec edn-value2))
#2021-07-2116:47mike_ananev@UK0810AQ2 yeah! This code reproduces the bug. @ikitommi do you have any suggestions?#2021-07-2116:49Ben SlessNow I can debug and investigate#2021-07-2116:49mike_ananevThe bug is when we have optional attribute in a map then process edn -> encode -> json -> decode ->edn2 is broken, edn != edn2 . This behavior is in version 0.6.0-20210721.071739-2. In 0.5.1 version all is working as expected.#2021-07-2117:42Ben SlessManaged to narrow the example down a bit:
(do
(def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
(def my-spec1 [:map
[:a/c1 {:optional true} :uuid]
[:a/c2 :uuid]])
(def s (m/schema my-spec1))
(def my-spec (m/schema [:map [:x s]]))
(def g (mg/generator my-spec))
(def edn-value (mg/generate g {:seed 9999}))
(def e (m/encoder my-spec custom-json-transformer))
(def encoded-value (e edn-value))
(def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
(def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
(def d (m/decoder my-spec custom-json-transformer))
(def edn-value2 (d encoded-value2))
(m/validate my-spec edn-value2))#2021-07-2117:42Ben SlessThis returns false consistently#2021-07-2118:26Ben SlessI managed to find the commit which caused the change:
9dc8da7a0649ed93554c1fe16ea48c2bebd724ad
fast collection transformers#2021-07-2118:32Ben SlessI am not sure where the bug is, but changing the implementation of -map-transformer to
(reduce
-comp
(map
(fn [[k t]]
(fn [x]
(if-let [e (.entryAt x k)]
(.assoc x k (t (.val e)))
x)))
ts))
Fixes it#2021-07-2118:32mike_ananev👍#2021-07-2207:51Ben Slesshttps://github.com/metosin/malli/issues/480#2021-07-2208:54ikitommithanks @UK0810AQ2. I hate it when there is test to cover this. My bad. The minimal repro:
(deftest regression-480-test
(let [value {:b #uuid"f5a54a8f-7d78-4495-9138-e810885d1cdb"}
schema [:map [:a :int] [:b :uuid]]]
(is (= value
(as-> value $
(m/encode schema $ mt/string-transformer)
(m/decode schema $ mt/string-transformer))))))#2021-07-2209:00ikitommiupdated clojars;
➜ ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.6.0-SNAPSHOT"}}}'
Downloading: metosin/malli/0.6.0-SNAPSHOT/malli-0.6.0-20210722.085801-3.pom from clojars#2021-07-2209:03ikitommibare minmial repro btw was:
(m/decode
[:map [:a :int] [:b :int]]
{:b "1"}
mt/string-transformer)
; => {:b "1"}#2021-07-2210:37mike_ananev@ikitommi @UK0810AQ2 thank you! Now, all test are passed. We will try to take version 0.6.0-20210722.085801-3 in our production.#2021-07-2015:27mike_ananevThis new behaviour of m/validate breaks some tests in my code.#2021-07-2015:28mike_ananevWhat behaviour is expected in 0.6.0 release? Should we adapt our tests for this new behaviour of m/validate ?#2021-07-2016:01gregI've just found out that mu/update-in can take a path that includes also catn options. It's so easy and concise to alter sequential schema with Malli! :grinning_face_with_star_eyes:#2021-07-2106:44Oliver GeorgeHaving my first look at Malli today. Is there a s/assert equivalent? This is a CLJS project. I'd like nice errors while developing and elided asserts in production.#2021-07-2107:15ikitommi@olivergeorge no, there is not. would it make sense to make the instrumentation work with cljs? it’s just clj ATM. If someone has mad skills at cljs-interop, clj-kondo emitting and malli.dev could be ported to cljs.#2021-07-2109:33Oliver GeorgeIt's all new to me but I'll keep it in mind as I get more familiar with Malli.#2021-07-2109:36Oliver GeorgeAdding a s/assert macro might be close to a copy/paste of cljs.spec.alpha/assert code. If it's considered useful/desirable.#2021-07-2109:54alpoxI was looking for an s/assert equivalent as well - me in Clojure though, not CLJS#2021-07-2109:56alpoxSo from the little hint to malli.dev i gather there is some similar functionality in there? I guess I'll have to look at it a bit closer#2021-07-2212:22David LevinovHey guys, very quick and simple question, I want to combine two schemas to validate for case a and case b, and I’d rather not involve regexes, and have one of the validation schemas to have the values not be of a certain set.
How would I rewrite this in a correct manner?
(def NotABC
(m/schema
[:map
[:field [:not [:enum "a" "b" "c"]]]]))
Thanks!#2021-07-2215:17ikitommilooks legit to me.#2021-07-2215:53David Levinov@ikitommi that’s exactly what I thought as well, but in the REPL:
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :not}
thanks for the response 👌#2021-07-2215:59ikitommioh, what. Try updating the malli dependency.#2021-07-2216:13David Levinovwelp, that does indeed solve the issue, thanks haha#2021-07-2214:11Lucy Wang@ikitommi Looks like the new function instrumentation only works for clj, but not for cljs yet. E.g. mi/instrument! and dev/start would throw when I use them in cljs. Is there any plan to add cljs support?#2021-07-2214:46ikitommi@wxitb2017 help most welcome on cljs-support.#2021-07-2215:14ikitommihttps://github.com/metosin/malli/issues/482#2021-07-2302:16Lucy WangCool!#2021-07-2219:14Ben SlessInitial attempt at writing a malli based parser for EDN datalog syntax
https://gist.github.com/bsless/632b4040a2b2ad7469369f52cd610c06
Can't figure out why ::clause breaks and it's getting late here. Feel free to poke at the code, suggest improvements, feedback, etc.#2021-07-2314:53Ben SlessDirect translation to spec works for parsing the forms, insights as to why it didn't work with malli most welcome, updated the gist#2021-07-2314:53Ben SlessDirect translation to spec works for parsing the forms, insights as to why it didn't work with malli most welcome, updated the gist#2021-07-2313:18robert-stuttaforddoes #malli work with #babashka ? (just did a demo for my team and our infra human is curious)#2021-07-2317:09borkdudemalli currently doesn't work (from source) in babashka#2021-07-2317:09borkdudewe have an issue here: https://github.com/babashka/babashka/discussions/906 to consider including it#2021-07-2317:26borkdude@U0509NKGK there are alternatives which currently work, such as spartan.spec (a spec drop-in replacement) and minimallist#2021-07-2317:27borkdudeif you could tell me how exactly malli would be useful for infra, please discuss in the issue, it could be an argument to include it#2021-07-2915:55robert-stuttafordtruly just pure curiosity at this point 🙂#2021-07-2314:58Ben SlessIs there a reason :ref schemas just die for regex parsing? Why do they have to be surrounded by :maybe?#2021-07-2315:20ikitommi@ben.sless refs directly in sequence schemas are disallowed, rationale: https://github.com/metosin/malli/blob/master/src/malli/impl/regex.cljc#L1-L34#2021-07-2315:21ikitommiyou can wrap anything in :schema to push it outside of the sequence.#2021-07-2316:19Ben SlessIt would be good to add in the readme that recursive seqexp reference schemas have to have another level of indirection via either maybe or schema#2021-07-2411:36ikitommidoc PRs always welcome#2021-07-2315:22ikitommiHere's an example with hiccup, which is recursive:
(def Hiccup
[:schema {:registry {"hiccup" [:orn
[:node [:catn
[:name keyword?]
[:props [:? [:map-of keyword? any?]]]
[:children [:* [:schema [:ref "hiccup"]]]]]]
[:primitive [:orn
[:nil nil?]
[:boolean boolean?]
[:number number?]
[:text string?]]]]}}
"hiccup"])
(def parse-hiccup (m/parser Hiccup))#2021-07-2315:33Ben SlessI'll try. I got the parser working besides ref which I just hacked around for now#2021-07-2316:15Ben SlessIt works!#2021-07-2316:16Ben SlessUpdated the gist with working version, next step is a dynamic interpreter, followed by a compiler.
It's pretty cool I can use the library to define new syntax. It extends itself#2021-07-2422:57nivekuilis there a way to work with bigints in cljs? trying to coerce a string (reitit route) to one#2021-07-2504:05nivekuilbigints are a headache with transit, just ended up using Long
(defn -string->bigint [x]
(if (string? x)
(try
#?(:clj (Long/parseLong x)
:cljs (let [x' (js/parseInt x)]
(if (> x' js/Number.MAX_SAFE_INTEGER)
(goog.math.Long/fromString x 10)
x')))
(catch #?(:clj Exception, :cljs js/Error) _ x))
x))
(defn string-transformer []
(mt/transformer
{:name :string
:decoders (assoc (mt/-string-decoders)
:i64 -string->bigint)
:encoders (assoc (mt/-string-encoders)
:i64 mt/-any->string)}))
#2021-07-2510:28Ben SlessIn providing a custom registry to a schema schema, what gets compiled? when?#2021-07-2913:37ikitommiugh. can’t recall. you need to follow the code.#2021-07-2622:28greg@ikitommi do you expect introducing :andn? In my schema some of the properties have relations that could be verified. I do it by adding a top level :and and then modify/navigate it using indices (0 for the schema, 1... for predicates). Not a bit deal to use indices, still, the case looks similar to catn .#2021-07-2913:36ikitommisure, PR welcome.#2021-07-2708:53Ben SlessI think this level of understanding the JIT is slightly above my pay grade.
If someone here has expertise on the subject it would be most welcome
https://github.com/metosin/malli/pull/485#2021-07-2718:00gregI've added a tip demonstrating that transformer, the one that calculates default value based on given fn in :default-fn
PR: https://github.com/metosin/malli/pull/487#2021-07-2801:30greg(def Schema (m/schema [:map
[:y {:limit 20} int?]]))
Given the above schema schema:
1. How to access properties of :y?
I tried
2. How to update :x to double it, e.g by passing #(* % 2)?
m/update combined with m/update-properties doesn't work for the same reason as above.
3. How to convert [:y {:limit 20} int?] to [:y {:limit 20} [:and int? [:> 0]] without loosing properties.
(mu/update Schema :y (fn [s] [:and s [:> 0]])) ;; => [:map [:y [:and int? [:> 0]]]]
; I expected: [:map [:y {:limit 20} [:and int? [:> 0]]]]
Calling the update removes the original properties.
Some workaround I think would be writing custom get & update based on m/children. Doesn't look a difficult taks, just thought about asking before reinventing a wheel 🙂#2021-07-2808:48TuomasI had a similar issue and went with an approach like
(def Schema (m/schema [:map
[:y [int? {:limit 20}]]]))
(m/validate Schema {:y "1"})
(m/validate Schema {:y 1})
(m/properties (mu/get Schema :y))
But I'm a novice and not really sure how this should be done#2021-07-2810:13gregI think it would work for some cases, but it might be not a good idea to store child-owned props at schema level.
Let's say I got this Schema
(def Schema (m/schema [:map
[:x number?]
[:y [number? {:default 0}]])
If I do the simple :y update to ensure its >= 0 I will get this:
(-> Schema
(mu/update :y (fn [s] [:and s [:>= 0]])))
; => => [:map
[:x number?]
[:y [:and [number? {:default 0}] [:>= 0]]]]
And I think this kind of properties, to work properly, they need to be kept in top level of schema for given child, e.g.:
[:map
[:x number?]
[:y [:and {:default 0} number? [:>= 0]]]
So to make this work you need both:
• moving props from child (`[:y]`) to schema (`[number?]`)
• adding custom update anyway (to move properties from original schema (`[number?]`) to a new enclosing schema (`[:and]`)
But when doing this properties maneuver, how do you know which are owned by child and which by schema? You don't unless you explicitly specify it e.g. in update method.
Conceptually I think it makes more sense to keep child-owned properties at child level, and schema-owned properties at schema level.#2021-07-2810:27gregActually I went this path, and I implemented this custom update for properties manouver when updating a schema (e.g. moving from [number?] to [:and] ), but as I wrote, I don't think it is good idea. As I think now, it really makes more sense to just add a few new utility methods: get-child, get-child-props, update-child, update-child-props, update-child-schema.
It is a bit weird that child properties disappear on child's schema update. It might be a bug. Who is John Galt? :man-shrugging:#2021-07-2811:33gregI wrote a few utils fns to sort out my problems with manipulating map schemas:
(defn get-child [s k] (->> (m/children s) (filter #(= k (first %))) (first)))
(defn get-child-props [s k] (-> (get-child s k) second))
(defn get-child-schema [s k] (-> (get-child s k) (nth 2)))
(defn update-child [s key f & args]
(let [new-children (->> (m/children s)
(map (fn [child]
(let [[k _p _v] child]
(if (= key k) (apply f child args) child)))))]
(m/into-schema (m/type s) (m/properties s) new-children)))
(defn update-child-props [s k f & args]
(update-child s k (fn [c]
(let [[k p v] c]
[k (apply f p args) v]))))
(defn update-child-schema [s k f & args]
(update-child s k (fn [c]
(let [[k p v] c]
[k p (apply f v args)]))))
(comment
(def Schema (m/schema [:map [:y {:default 20} int?]]))
(get-child-schema Schema :y) ;; => int?
(get-child-props Schema :y) ;; => {:default 20}
;; updates of child props
(update-child-props Schema :y update :default dec) ;; => [:map [:y {:default 19} int?]]
(update-child-props Schema :y assoc :limit 50) ;; => [:map [:y {:default 20, :limit 50} int?]]
;; update of child schema preserving child props
(update-child-schema Schema :y (fn [s] [:and s [:> 0]])) ;; => [:map [:y {:default 20} [:and int? [:> 0]]]]
;; update of child schema removing the props
(mu/update Schema :y (fn [s] [:and s [:> 0]]))) ;; => [:map [:y [:and int? [:> 0]]]]
what do you think?#2021-07-2913:31ikitommiLooks good. Few comments:
• mu/find gets the entry:
(def Schema (m/schema [:map [:y {:limit 20} int?]]))
(mu/find Schema :y)
; => [:y {:limit 20} int?]
• when calling m/into-schema you should always use m/parent instead of m/type as first argument - m/type makes a registry lookup, m/parent points directly to the IntoSchema instance that created the schema. Faster AND adds support for non-registered schemas, e.g.
(def Over6
(m/-simple-schema
{:type :user/over6 ;; dummy-type, not registered
:pred #(and (int? %) (> % 6))
:type-properties {:error/message "should be over 6"
:decode/string mt/-string->long
:json-schema/type "integer"
:json-schema/format "int64"
:json-schema/minimum 6
:gen/gen generate-over6}}))
(= Over6 (m/parent [Over6 {:json-schema/example 42}]))
; => true
• for now, we could have a mu/-update-entry-properties helper, PR welcome
• for future (1.0.0), the entry design should be revisited, not happy how it is now#2021-07-2914:31gregthanks a lot for the comments, I'll prepare a PR when I find spare time#2021-07-2810:37Ben SlessFigured out today a decoder can be used to reject wrong values: {:compile (fn [schema _] (let [s (set (m/children schema))] (fn [v] (s v))))}#2021-07-2913:36ikitommibit like mt/strip-extra-keys-transformer 😉#2021-07-3008:54Ben SlessSimilar, but just "forces" invalid enum values to be nil.#2021-07-2810:37Ben SlessSort of coercion#2021-07-2811:33gregI wrote a few utils fns to sort out my problems with manipulating map schemas:
(defn get-child [s k] (->> (m/children s) (filter #(= k (first %))) (first)))
(defn get-child-props [s k] (-> (get-child s k) second))
(defn get-child-schema [s k] (-> (get-child s k) (nth 2)))
(defn update-child [s key f & args]
(let [new-children (->> (m/children s)
(map (fn [child]
(let [[k _p _v] child]
(if (= key k) (apply f child args) child)))))]
(m/into-schema (m/type s) (m/properties s) new-children)))
(defn update-child-props [s k f & args]
(update-child s k (fn [c]
(let [[k p v] c]
[k (apply f p args) v]))))
(defn update-child-schema [s k f & args]
(update-child s k (fn [c]
(let [[k p v] c]
[k p (apply f v args)]))))
(comment
(def Schema (m/schema [:map [:y {:default 20} int?]]))
(get-child-schema Schema :y) ;; => int?
(get-child-props Schema :y) ;; => {:default 20}
;; updates of child props
(update-child-props Schema :y update :default dec) ;; => [:map [:y {:default 19} int?]]
(update-child-props Schema :y assoc :limit 50) ;; => [:map [:y {:default 20, :limit 50} int?]]
;; update of child schema preserving child props
(update-child-schema Schema :y (fn [s] [:and s [:> 0]])) ;; => [:map [:y {:default 20} [:and int? [:> 0]]]]
;; update of child schema removing the props
(mu/update Schema :y (fn [s] [:and s [:> 0]]))) ;; => [:map [:y [:and int? [:> 0]]]]
what do you think?#2021-07-2913:31ikitommiLooks good. Few comments:
• mu/find gets the entry:
(def Schema (m/schema [:map [:y {:limit 20} int?]]))
(mu/find Schema :y)
; => [:y {:limit 20} int?]
• when calling m/into-schema you should always use m/parent instead of m/type as first argument - m/type makes a registry lookup, m/parent points directly to the IntoSchema instance that created the schema. Faster AND adds support for non-registered schemas, e.g.
(def Over6
(m/-simple-schema
{:type :user/over6 ;; dummy-type, not registered
:pred #(and (int? %) (> % 6))
:type-properties {:error/message "should be over 6"
:decode/string mt/-string->long
:json-schema/type "integer"
:json-schema/format "int64"
:json-schema/minimum 6
:gen/gen generate-over6}}))
(= Over6 (m/parent [Over6 {:json-schema/example 42}]))
; => true
• for now, we could have a mu/-update-entry-properties helper, PR welcome
• for future (1.0.0), the entry design should be revisited, not happy how it is now#2021-07-2909:43danielnealis there an malli.util/assoc that will allow to set the properties and schema of a key#2021-07-2909:43danielnealor a different function?#2021-07-2910:04gregThat was part my question I posted https://clojurians.slack.com/archives/CLDK6MFMK/p1627435833075200. I did not found a way of doing it at Malli.
But it is fairly easy to do using m/children.
If you have a look at my other https://clojurians.slack.com/archives/CLDK6MFMK/p1627471982078700?thread_ts=1627435833.075200&cid=CLDK6MFMK on this channel, I wrote a few utility methods.
One of them is update-child-props.
It was a first draft, I modified them since then, but even with it, you can do what you want, e.g.
(update-child-props (m/schema [:map [:y {:default 20} int?]]) :y update :default dec)
;; => [:map [:y {:default 19} int?]]
(update-child-props (m/schema [:map [:y {:default 20} int?]]) :y assoc :limit 50)
;; => [:map [:y {:default 20, :limit 50} int?]]
#2021-07-2912:57gregUnless I misunderstood "allow to set the properties and schema of a key" 😅#2021-07-2913:34ikitommimaybe:
(mu/assoc
[:map [:y {:old true} int?]]
[:y {:new true}] string?)
; => [:map [:y {:new true} string?]]#2021-07-2914:28danielnealaha, thanks!#2021-07-2913:43ItayHi,
I posted this in the #reitit channel but still did not get any response.
No reply on the issue I opened either.
Since it is Malli related, I thought perhaps I could get some feedback here.
Thanks.#2021-07-2913:44ikitommi@itaysabato I can take a look at that today/tomorrow#2021-08-0208:22ItayGood morning 🙂
Any news regarding this issue?#2021-08-0213:02ikitommitotally forgot, sorry. will try to look later today.#2021-08-0306:01ikitommidid a PR, might fix it. Depending on the exceptations#2021-08-0308:17ItayThank you for this!
Looks like it did.
I commented on one line that I didn't get, but the rest looks solid to me.#2021-08-0309:27ikitommianswered. I'm not near computer, if you could verify the old behavior (does it strip extras or not), would help. Thanks.#2021-08-0810:34ItaySorry, I was away for a few days.
Is it still relevant to test the old strip keys behavior?
I will soon try the fixed version#2021-07-2913:45Itay@ikitommi That would be lovely, thank you!#2021-07-2913:55ikitommiquick poke on instrumentation: malli.dev/start! will now collect function schemas from all public vars from all loaded namespaces.#2021-07-2913:56ikitommiI would like to add Var-watches too, so any Var change would re-trigger instrumentation of that Var. But didn’t work in the 30min timebox I had for it.#2021-07-2913:57ikitommianyway, dx should be better than before. feedback welcome on the instrumention. If anyone has tried that…#2021-07-2913:57ikitommi#2021-07-2914:02ikitommigoal is to do the first version of pretty error reporting for 0.6.0, to celebrate the two year birthday of the initial demo of that (https://github.com/metosin/malli/issues/19). Making open source 1h here and there isn’t the fastest way to finish stuff 😎 #2021-07-2914:02Noah Bogarthey all, just started using malli in a new part of my project, and i’m struggling to check if a value is a function#2021-07-2914:03Noah Bogart(def Step
[:map {:closed true}
[:uuid uuid?]
[:continue-fn fn?]
[:complete? boolean?]
[:phase keyword?]])
#2021-07-2914:03ikitommishould work#2021-07-2914:03ikitommifn? was added few days ago thou, it was missing. you should depend on the latest sha#2021-07-2914:04Noah Bogartah that’s probably it, i’m using the clojars version#2021-07-2914:04ikitommi[:fn fn?] works also. :fn is the escape hatch, “any function”#2021-07-2914:07Noah Bogartthat works! thanks so much#2021-07-2914:16ikitommiimmutable clojars-snapshot of the latest stuff:
➜ ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.6.0-SNAPSHOT"}}}'
Downloading: metosin/malli/0.6.0-SNAPSHOT/malli-0.6.0-20210729.141459-4.pom from clojars#2021-07-2914:17ikitommibtw, @nbtheduke there is a new guide for function schmemas at https://github.com/metosin/malli/blob/master/docs/function-schemas.md#2021-07-2914:18Noah Bogartmissed that there were further docs, i’ll read over that now#2021-07-3010:05Ben SlessCurious why change from ArrayList to LinkedList in the transformers' implementation. Aren't arrays better for iteration and cache?#2021-07-3010:26ikitommiLinkedList has much simpler iterator impl. Didn't see any visible perf difference thou.#2021-07-3011:05Ben SlessThis is slightly theoretical, but ArrayLists have a smaller memory footprint and their backing array is contiguous in memory which ought to give better performance. Also, although the iterator implementation is simpler, ArrayLists would probably benefit from branch predictions, while pointer-chasing with linked lists gives you no guarantees#2021-07-3011:06Ben Slesshttps://stackoverflow.com/questions/8436489/using-a-linkedlist-or-arraylist-for-iteration#2021-07-3011:07Ben SlessModern CPUs like arrays 😞#2021-07-3015:36gregI guess you talk about https://github.com/metosin/malli/commit/dd19b9e.
I, same as @UK0810AQ2, would choose ArrayList, especially that we know the size of data. Theoretically it should be faster in allocation of memory and further iteration, but well, always good to see relevant data.
Personally I don't care about performance that much so I don't mind either of them. Still, I'm curious, @U055NJ5CC why the iterator's impl matters to you since its interface is the same?#2021-07-3016:04ikitommiCollection transformers will be rewritten by @UK0810AQ2 to remove all iteration. But, perf comes from impl, not perf. Was really iteresting read, I still think the iteration is faster on LL, but, irrelevant here. As a generic data structure AL is much better.#2021-07-3017:51Ben SlessI don't know if the composition based solution will be faster. Will try to poke at it with jmh#2021-07-3111:39Ben SlessRight, jmh is running, results in a few hours#2021-07-3113:56Ben Sless([:map-transformer/linked-list {:count 1} [6.156812627775496E7 "ops/s"]]
[:map-transformer/linked-list {:count 2} [3.2994990990360465E7 "ops/s"]]
[:map-transformer/linked-list {:count 4} [1.7037915234624855E7 "ops/s"]]
[:map-transformer/linked-list {:count 8} [7557295.433684901 "ops/s"]]
[:map-transformer/linked-list {:count 16} [1566793.27656629 "ops/s"]]
[:map-transformer/linked-list {:count 32} [704492.6211734915 "ops/s"]]
[:map-transformer/array-list {:count 1} [6.3440679176932156E7 "ops/s"]]
[:map-transformer/array-list {:count 2} [3.439043251403162E7 "ops/s"]]
[:map-transformer/array-list {:count 4} [1.689684037729955E7 "ops/s"]]
[:map-transformer/array-list {:count 8} [7481296.308118063 "ops/s"]]
[:map-transformer/array-list {:count 16} [1550320.2392137954 "ops/s"]]
[:map-transformer/array-list {:count 32} [713659.2962506406 "ops/s"]]
[:map-transformer/comp {:count 1} [8.510126517455931E7 "ops/s"]]
[:map-transformer/comp {:count 2} [4.2607291886022285E7 "ops/s"]]
[:map-transformer/comp {:count 4} [1.9971178945141207E7 "ops/s"]]
[:map-transformer/comp {:count 8} [8451028.653528532 "ops/s"]]
[:map-transformer/comp {:count 16} [1599471.9346010373 "ops/s"]]
[:map-transformer/comp {:count 32} [737048.961960025 "ops/s"]])
#2021-07-3114:41Ben SlessAnother avenue I want to explore is creating from an array. Can't avoid an array copy anyway, so why not copy just once?#2021-07-3119:24Ben Slessmore plausible to achieve with a tuple transformer, don't know if can be done with map#2021-08-0105:32Ben SlessI think I found a faster way to transform a tuple:
(defn -tuple-transformer [ts]
(let [ts (map (fn [[k t]] (let [i (int k)] (fn [^objects arr] (aset arr i (t (aget arr i))) arr))) ts)
t (apply -comp ts)]
(fn [^clojure.lang.APersistentVector x]
(clojure.lang.LazilyPersistentVector/createOwning (t (.toArray x))))))
Will check some arities combinations, initial tests look promising#2021-07-3011:25danielnealIf you want to have different error messages for different predicates in a malli schema, would it be wrong to do this#2021-07-3011:25danielneal[:and [:fn {:error/message "error 1"} some-constraint]
[:fn {:error/message "error 2"} some-other-constraint]]#2021-07-3013:56danielnealwhat’s the schema for a malli schema? e.g. for a key in a map there’s a key in a map that can be a schema [:map [:a int?] [:spec :???schema???]]#2021-07-3014:35respatializedThe term I've used to describe this is alternately "meta-schema" or "self-describing schema" - there isn't one yet for malli schemas.#2021-07-3014:43danielnealah ok, thanks#2021-07-3014:44danielnealI’ll just use any? I guess#2021-07-3013:56danielnealcouldn’t figure out what to search for#2021-07-3013:57danielnealthe word schema appears a lot 😉#2021-07-3014:03danielnealis it just malli/schema??#2021-07-3014:04danielnealhm no that doesn’t seem to work#2021-07-3014:13Noah Bogartdoes malli care about stuff like volatile!? I have a schema for a map and i’m thinking of changing it to a volatile, so how do I put that in my schema? I changed the schema to [:volatile [:map …]] but that didn’t work#2021-07-3014:35respatializedThe term I've used to describe this is alternately "meta-schema" or "self-describing schema" - there isn't one yet for malli schemas.#2021-08-0108:31Ben Slesshot take: :fn schema should be deprecated#2021-08-0108:49ikitommiwhy is that?#2021-08-0109:12Ben Sless95% of the instances I saw it used the programmer should have used properties or defined a -simple-schema#2021-08-0109:14Ben SlessIt is used in a way too general manner, usually one which impairs serialization and generation#2021-08-0109:15Ben SlessWe can write guides and improve documentation and give workshops and do backflips and pay for ads on the superbowl, people will still use :fn because it's easy#2021-08-0109:20Ben Slessgenerators and transformers suffer from this, big time#2021-08-0109:21Ben SlessAnd it makes the code opaque#2021-08-0110:56Ben SlessI did say this was a hot take 🙃#2021-08-0111:20gregI think -simple-schema is not always what you want to do. And yes :fn is pretty handy, especially if you don't need transformers and generators but only validation. I'm not sure it would be a good move to force people to use -simple-schema just for validation (e.g. involving two or more map entries).#2021-08-0111:40Ben SlessThis is absolutely me speaking from frustration, not a definitive or proscriptive solution#2021-08-0111:45Ben SlessThe problem isn't using fn but using it where [:string {:max 10 }] would do#2021-08-0111:50Ben SlessI also see predicate functions used a lot where decoders would be better#2021-08-0114:56gregOk, make total sense now. I just took it as a proposal of change. I need to develop a sense of humour I guess 😅#2021-08-0115:08Ben SlessSee the "hot take" prefix#2021-08-0115:37gregI missed that :face_palm:😅#2021-08-0115:10Ben SlessFrustrated by half a day fighting with badly written and used malli schemas
Everything can and will be abused#2021-08-0115:41gregI don't know you particular case, so just guessing... Maybe organising some internal training showing how to use Malli could avoid that kind of tool misuse in the future 😉🚀#2021-08-0116:28Ben SlessI am actually working on internal training which I hope I'll be able to release mostly as a contributed getting started guide#2021-08-0115:13Noah BogartI suspect this also has to do with docs. It can be hard to fully enumerate the possibilities of a given API, and something like :fn just works. #2021-08-0115:36gregI agree with this point.
@UK0810AQ2 tbh with you, I understand the points that frustrates you, but I wouldn't be two weeks ago. The thing is that Malli is nice to use once you know it, but it is fairly confusing when you start. I think that it might be the source of the problem, that people are using the simplest solutions they know, because they simply don't know all the features of the tool.
Let's take my example. I used spec before, but haven't used spec.tools nor prismatic schema. Some of the concepts were simply new to me. E.g. transformers, there are at least four ways of transforming values with Malli, and it is not obvious what they are and how they work. I discovered it by reading the docs, examples, Malli impl and reading the Tommis answers here. I don't have daily job and I could invest a bit of time, still not everyone can afford that much of investment, so most of the time, I guess, people pragmatically makes things done 🙂
In this particular topic, using Malli transformers, I intend to make a comprehensive writing at some point to make the process easier for others. I think in case of :fn and simple-schema case is the same, people might not be aware when/where to use what 🙂#2021-08-0116:30Ben SlessI agree. Malli's docs are comprehensive but like man pages they don't tell you where to start, they don't guide you towards "I want to do X"#2021-08-0116:14zxspectrris there a more idomatic way of defining a map that must have a key but the value can be nullable. Here's what I've currently got:
(defn ? [predicate] [:or predicate nil?])
(def MyThing
[:map
[:someValue (? string?)]])#2021-08-0116:15zxspectrrOnly reason I want to force the existence of a key so it appears in a JSON response with null as it's value#2021-08-0116:51alpoxI'm not 100% sure but would that not be [:someValue [:maybe string?]]#2021-08-0117:04zxspectrroh, I didn't realise there's a [:maybe]#2021-08-0117:04zxspectrrnice#2021-08-0117:04zxspectrryep that works, ty 🙂#2021-08-0117:05zxspectrrmy experience with malli so far has been quite positive#2021-08-0213:05ikitommiwip#2021-08-0218:54Ben SlessIs it integrated with kondo as well?#2021-08-0219:51ikitommimalli.dev is, the red underline & bump are from kondo. The pretty printer is currently just for runtime errors.#2021-08-0213:06ikitommithe error formatters use fipp, look like this:
(defmethod v/-format ::m/invalid-input [_ _ {:keys [args input]} printer]
{:body
[:group
(-block "Invalid function arguments:" (v/-visit args printer) printer) :break :break
(-block "Input Schema:" (v/-visit input printer) printer) :break :break
(-block "Errors:" (-errors input args printer) printer) :break :break
(-block "More information:" (v/-color :link "" printer) printer)]})#2021-08-0213:08ikitommithere will be pretty/reporter to just prints the errors and pretty/thrower to throw ’em.#2021-08-0213:13Noah Bogartthat looks great#2021-08-0213:13Noah Bogartis there a better idiom for this?
(defn Step? [step]
(if (m/validate Step step)
step
(throw (Exception.
(->> step
(m/explain Step)
(me/humanize))))))
#2021-08-0213:19danielnealhaha I literally wrote that exact function 5 mins ago and was wondering the same thing#2021-08-0213:57ikitommi• you could create a validator & explainer before, if perf matters
• I recommend throwing ex-imfo with :type , easier to catalog the errors
• humanize can fail in some corner cases, should be robust, fixes welcome
• there could be a helper for this in the future in malli, using m/-fail! which works well with the upcoming pretty printer#2021-08-0214:02ikitommiMalli internal errors just throw the minimum set of data (to be fast), to be post-processed when rendering the error. e.g. explain and humanize are pure, can be done on rendering, if the value & schema are present at ex-data. Might not be relevant on actual projects.#2021-08-0214:04ikitommiI haven't heard of a robust way to override the repl default exception handler (from the repl) - would allow pretty errors for all Clojure: just throw data and add custom renderers for all :types & exception classes#2021-08-0214:06Noah Bogartso something like this?
(defn SimpleStep? [step]
(if (validate-step @step)
step
(throw (ex-info "Step isn't valid" {:type (explain-step @step)}))))
#2021-08-0214:07Noah Bogartgood point about the validator and explainer, thanks#2021-08-0218:58Ben SlessI'm leaning towards this form:
(defn needs-a-good-name
[s]
(let [s (m/schema s)
v (m/validator s)
d (m/decoder s ,,,) ;; optional?
e (m/explainer s)]
(fn [x]
(let [x (d x)]
(if (v x)
x
(throw (ex-info "invalid something" (e x))))))))
More generally, you can pass two functions, on-success and on-fail, then you can decide if you want to throw or drop or invalid data#2021-08-0218:59Ben SlessSomething like
(defn needs-a-good-name
[s respond raise]
(let [s (m/schema s)
v (m/validator s)
d (m/decoder s ,,,) ;; optional?
e (m/explainer s)]
(fn [x]
(let [x (d x)]
(if (v x)
(respond x)
(raise e x))))))
#2021-08-0311:03gregI think these two examples could be posted somewhere#2021-08-0311:13Ben SlessI'll clean it up a bit and PR it to the tips?
Maybe we should add a cookbook#2021-08-0219:47ikitommicoercer? did you mean (raise (e x))? Look good anyway 👍 👍 👍#2021-08-0220:35Ben Slesscoercer is probably good. Wasn't sure about passing the original value or just the report to raise#2021-08-0308:01vlaaadHi! Is there a malli equivalent of s/keys* ?#2021-08-0309:18ikitommiNot yet, I guess the could be...#2021-08-0308:07vlaaadside note:
(let [s [:catn [:rest [:? any?]]]]
(m/unparse s (m/parse s [])))
=> [nil] ;; should be []
#2021-08-0309:18ikitommiOh, it's a bug, issue /& PR welcome!#2021-08-0308:54Ben SlessAny idea how to make the transformer tests not rely on map order?
https://github.com/metosin/malli/pull/497#2021-08-0309:17ikitommi@ben.sless looks great! Commented the PR, ideas about the order & stuff#2021-08-0315:20Noah Bogarthow do i do this?
(def PhaseStepSchema
(mu/merge
BaseStepSchema
[:map {:closed true}
[:phase qualified-keyword? {:namespace :phase}]]))#2021-08-0315:21Noah Bogarti want the :phase value to be a namespace-qualified keyword, with the namespace :phase , aka :phase/action#2021-08-0316:44Noah Bogartsolution i found: [:phase [:qualified-keyword {:namespace :phase}]]#2021-08-0317:42Ben SlessI came across an extremely weird phenomenon which I don't think should happen
1. Unhandled java.lang.VerifyError
Stack map does not match the one at exception handler 1251 Exception
Details: Location:
malli/core_test$fn__30148$fn__32525.invoke()Ljava/lang/Object; @1251:
astore_3 Reason: Type 'java/lang/Throwable' (current frame,
locals[6]) is not assignable to 'clojure/lang/Keyword' (stack map,
locals[6]) Current Frame: bci: @1175 flags: { } locals: {
'malli/core_test$fn__30148$fn__32525',
'clojure/lang/IPersistentVector', null, null, 'java/lang/Object',
null, 'java/lang/Throwable' } stack: { 'java/lang/Throwable' }
Stackmap Frame: bci: @1251 flags: { } locals: {
'malli/core_test$fn__30148$fn__32525',
'clojure/lang/IPersistentVector', 'java/lang/Object', null,
in accumulating errors #84
Will try another JVM version#2021-08-0317:43Ben SlessThis happens in the same test, on both Java 15 and 11, only after I run it twice#2021-08-0317:44Ben SlessHappens on java 8, too#2021-08-0409:11Ben SlessI managed to narrow it down to multi-schemas#2021-08-0409:12Ben SlessSpecifically, the last bit#2021-08-0409:12Ben Sless(let [schema [:multi {:dispatch first}
[:human [:cat [:= :human]]]
[:bear [:cat [:= :bear] [:* :int]]]
[::m/default [:tuple :string :string]]]]
(testing "validate"
(is (m/validate schema [:human]))
(is (m/validate schema [:bear 1 2 3]))
(is (m/validate schema ["defaultit" "toimii"]))
(is (not (m/validate schema [:so :invalid]))))
(testing "explain"
(is (not (m/explain schema [:human])))
(is (not (m/explain schema [:bear 1 2 3])))
(is (not (m/explain schema ["defaultit" "toimii"])))
(is (results= {:schema schema,
:value [:so :invalid],
:errors [{:path [::m/default 0], :in [0], :schema :string, :value :so}
{:path [::m/default 1], :in [1], :schema :string, :value :invalid}]}
(m/explain schema [:so :invalid]))))
(testing "parser"
(is (= (miu/-tagged :human [:human]) (m/parse schema [:human])))
(is (= (miu/-tagged :bear [:bear [1 2 3]]) (m/parse schema [:bear 1 2 3])))
(is (= (miu/-tagged ::m/default ["defaultit" "toimii"]) (m/parse schema ["defaultit" "toimii"])))
(is (= ::m/invalid (m/parse schema [:so :invalid])))))
#2021-08-0409:12Ben SlessSomething here is very wrong#2021-08-0409:13Ben SlessEven more specifically, it's the parser#2021-08-0409:14gregI've been using REBL recently and I noticed that Malli schemas are printed as regular one line strings, which is annoying.
Although I found it is possible to provide custom viewer to REBL. Example:
(defn schema? [s]
(m/schema? (try (m/schema s) (catch Exception _))))
(defn setup-rebl-viewers []
(require '[cognitect.rebl])
(cognitect.rebl/update-viewers
{:malli/schema {:pred #'schema?
:ctor (fn [v]
(-> (m/form (m/schema v))
(fipp.edn/pprint {:width 80})
(with-out-str)
(cognitect.rebl.renderers/string-code-viewer)))}}))
#_(setup-rebl-viewers)
This is just a simple pretty print, but we could apply other strategies for formatting.
If you have an idea what's the best way of formatting Malli schemas, please let me know in the comment.#2021-08-0409:15Ben Slessdatafy?#2021-08-0409:17Ben SlessExtend datafy to schemas and return form?#2021-08-0410:17gregHmm, I'll give it a try later :thinking_face:#2021-08-0410:17gregDo you datafy malli schemas? Could you post an example?#2021-08-0410:51Ben SlessBit of black magic:
(require '[clojure.core.protocols :as p]
'[malli.core :as m])
(extend-protocol p/Datafiable
Object
(datafy [x]
(cond
(satisfies? m/Schema x)
(do
(extend-protocol p/Datafiable
(class x)
(datafy [y] (m/-form y)))
(p/datafy x))
(satisfies? m/IntoSchema x)
(do
(extend-protocol p/Datafiable
(class x)
(datafy [y] (m/-form (m/schema y))))
(p/datafy x))
:else x)))
#2021-08-0413:55gregI think it could be simplified to
(extend-protocol p/Datafiable
Object
(datafy [x]
(cond
(satisfies? m/Schema x) (p/datafy (m/form x))
(satisfies? m/IntoSchema x) (p/datafy (m/form (m/schema x)))
:else x)))
Still as I mentioned in the other post (https://clojurians.slack.com/archives/C03S1KBA2/p1628084493261600?thread_ts=1628082589.258900&cid=C03S1KBA2), I've got a feeling extending Object is not a good idea 😬#2021-08-0501:01emccueor, add some metadata to your schemas that extends the protocol#2021-08-0501:02emccuethats probably the best way#2021-08-0501:04emccue(with-meta [:map [:x int?]] ^{`p/datafy (fn [this] (p/datafy (m/form this)))})#2021-08-1022:36gregThanks for posting example. Still I think that would be yet another extreme if I add my own m/schema to keep it everywhere...#2021-08-1022:42gregActually, maybe this should be added to Malli itself, to m/schema :thinking_face:
What do you think @U055NJ5CC about making Malli schemas datafy-able?
(it is just an idea, I'm not sure is it good or bad, i'm just trying to figure out best practices of using these new protocols for my own and in this very thread it stroke me Malli could be a good place to define that datafy behaviour)#2021-08-0409:55danielnealI’m trying to make a recursive schema#2021-08-0409:55danielnealdoing something like this (simplified)#2021-08-0409:55danielneal(malli/validate
[:vector {:registry {::property
[:map
[:property/type keyword?]
[:property/group {:optional true}
[:vector ::property]]]}}
::property]
[{:property/type :string}
{:property/type :key}])#2021-08-0409:55danielnealbut I get a stackoverflowerror#2021-08-0409:56danielnealI’m not sure model this kind of thing correctly#2021-08-0409:57danielnealoh I see it, i need :ref ?#2021-08-0410:06delaguardoyes#2021-08-0410:51Ben SlessBit of black magic:
(require '[clojure.core.protocols :as p]
'[malli.core :as m])
(extend-protocol p/Datafiable
Object
(datafy [x]
(cond
(satisfies? m/Schema x)
(do
(extend-protocol p/Datafiable
(class x)
(datafy [y] (m/-form y)))
(p/datafy x))
(satisfies? m/IntoSchema x)
(do
(extend-protocol p/Datafiable
(class x)
(datafy [y] (m/-form (m/schema y))))
(p/datafy x))
:else x)))
#2021-08-0517:04Ben SlessI forgot dealing with time was such a pain in the#2021-08-0615:55richiardiandreaHello there, I know this is not the right channel but I have a question about spec-tools
Is there out there a library for creating graphviz svg from a spec by any chance?#2021-08-0617:06ikitommiI recall there is. Haven't used thou.#2021-08-0622:45richiardiandreaok thanks I'll google it better, could not find it#2021-08-0807:04ikitommimaybe https://github.com/jebberjeb/specviz?#2021-08-0915:36richiardiandreaAh yeah I tried that one, no dice, I thought I saw a spec-tools specific one...however Malli looks super cool and I should try it out 😄
Thanks for the link anyways!#2021-08-0620:00ikitommiHumanized errors might finally work as expected. Fixes all known issues and is much simpler implementation. If someone can still crash it with some input, I’m all 👂s. Maybe we can now remove the safe-humanize from projects 🙂 https://github.com/metosin/malli/pull/502#2021-08-0620:03ikitommibig change is that the humanized form is taken from the first failure. e.g.
(-> [:map [:x [:and [:map [:y :any]] seq?]]]
(m/explain {:x {}})
(me/humanize))
;{:x {:y ["missing required key"]
; :malli/error ["should be a seq"]}}
(-> [:map [:x [:and seq? [:map [:y :any]]]]]
(m/explain {:x {}})
(me/humanize))
; {:x ["should be a seq"]}#2021-08-0620:31naomarikthat's awesome, really needed that!#2021-08-0620:03ikitommisame with sequences & sets.#2021-08-0620:05Noah Bogartdoes humanize work with records now?#2021-08-0620:05Noah Bogartwell, i guess i should say that that’s part of explain, not humanize so nvm lol#2021-08-0620:06ikitommirecords, good question, will test#2021-08-0620:08ikitomminope:
actual: java.lang.UnsupportedOperationException: Can't create empty: malli.error_test.Horror
at malli.error_test.Horror.empty (error_test.cljc:549)
clojure.core$empty.invokeStatic (core.clj:5236)
malli.error$_assoc_in.invokeStatic (error.cljc:147)#2021-08-0620:08Noah Bogarta simple (into {} rec) works for me right now#2021-08-0620:08ikitommiidea how to create empty records?#2021-08-0620:09Noah Bogarti don’t believe you can, lol#2021-08-0620:09Noah Bogarthttps://clojure.atlassian.net/browse/CLJ-1975#2021-08-0620:09Noah Bogartthat’s for spec but discusses the issue#2021-08-0620:14ikitommihttps://github.com/metosin/malli/pull/502/commits/19c3bbe34d2d0831ba6d4a3138debd36f3642267#2021-08-0620:14ikitommithe humanized for doesn’t preserve the Records, but then again, it forces all sequences to vectors. it’s for… humans.#2021-08-0620:15Noah Bogarthell yeah, thank you so much#2021-08-0620:24ikitommimerged in master#2021-08-0620:39ikitommimerged also the new pretty (schema error) printer into master. Not expound, but one could build such on top of this.#2021-08-0817:02ikitommiit’s out! (actually 0.6.1 as there was a missing dependency)#2021-08-0817:55emccueGetting what is perhaps a simple error, but#2021-08-0817:56emccue(ns dev.mccue.domain.user
(:require
[malli.instrument :as instrument]))
;; ----------------------------------------------------------------------------
(def User [:and [:map
[:user/email :string]
[:user/password-hash :string]]
[:fn {:error/message "should have user metadata"}
(fn [o] (= (type o) ::user))]])
;; ----------------------------------------------------------------------------
(defn create
{:malli/schema [:=>
[:cat [:map
[:email :string]
[:password-hash :string]]]
User]}
[{:keys [email password-hash]}]
^{:type ::user}
{:user/email email
:user/password-hash password-hash})
;; ----------------------------------------------------------------------------
(def ^{:malli/schema [:=> [:cat User] :string]}
email
:user/email)
;; ----------------------------------------------------------------------------
(def
^{:malli/schema [:=> [:cat User] :string]}
password-hash
:user/password-hash)
;; ----------------------------------------------------------------------------
(instrument/collect!)
;; ....
(malli.dev/start! {:report (malli.dev.pretty/reporter)})#2021-08-0905:03TuomasI could reduce it down to be about malli.dev.pretty/reporter and :fn but couldn't figure out the reason for overflow.
(defn identity-42
{:malli/schema [:=> [:cat [:fn (fn [n] (= 42 n))]] :int]}
[a] a)
(instrument/collect!)
(malli.dev/start! {:report (malli.dev.pretty/reporter)})
(identity-42 41) ; Execution error (StackOverflowError) at fipp.ednize/override? (ednize.clj:12)
(malli.dev/stop!)
(malli.dev/start!)
(identity-42 41) ; :malli.core/invalid-input {:input [:cat [:fn #funct ...#2021-08-0817:56emccuethis is what i load in (requiring malli dev and malli pretty in repl)#2021-08-0817:56emccue(password-hash {})
Execution error (StackOverflowError) at fipp.ednize/override? (ednize.clj:12).
null#2021-08-0817:57emccueand this is what happens when i try invalid input#2021-08-0817:57emccue(password-hash (create {:email "A" :password-hash "a"}))
=> "a"#2021-08-0817:57emccuevalid input works fine though#2021-08-0817:57emccuejust changed to [metosin/malli "0.6.1"]#2021-08-0910:21ikitommioh, it's the function values, need to short-circuit on them. Should be easy to fix.#2021-08-0910:25ikitommithough experiment:
• persist function schemas into edn/file (var->schema)
• write a clj-kondo plugin/hook that looks from afile if a Var has a malli schema defined. If it has runs that validation (inputs & outputs) to it and reports. Static analysis with full malli :thinking_face: 🚀 parrot #2021-08-0910:29Ben Slessseparating type and code for functions seems like a footgun waiting to go off#2021-08-0910:29Ben Slesssomeone will change one and not the other#2021-08-0910:30borkdudeI think you could have some kind of development runtime hook that writes clj-kondo type config as you go. But if I understand correctly malli already has this#2021-08-0910:31borkdudeone issue is that statically visible values are not the runtime values, so validating on those has different behavior#2021-08-0910:31ikitommiyes, malli.dev/start! re-writes the the clj-kondo config on any change to the function registry#2021-08-0910:31ikitommicould emit the new var->malli-schema file too at the same time.#2021-08-0910:32ikitommi(and malli.dev/stop! removes the file(s))#2021-08-0910:34ikitommiyes, that works with simple/demo cases. would need typedclojure to follow the types for real? and actual runtime to track the values for real?#2021-08-0910:36ikitommiI guess one could just add new keys to the clj-kondo config too? e.g. :malli/schema [:=> [:cat :int] :int]]#2021-08-0910:37ikitommiwould be just one config file then, always up-to-date.#2021-08-0910:38borkdudeyou mean type aliases?#2021-08-0910:40ikitommi{:linters
{:type-mismatch
{:namespaces
{malli.demo
{plus
{:arities
{1 {:args [:int]
:ret :int
:malli/schema [:=> [:cat int?] pos-int?]}}}}}}}}#2021-08-0910:41ikitommi… for the plugin to read from.#2021-08-0910:44borkdudebut when your plugin is called with the input types, it would have to do something similar to the clj-kondo "type" system right#2021-08-0910:45borkdudeso why not convert the schema to the clj-kondo type system immediately#2021-08-0911:18ikitommiit is converted already to clj-kondo type system. In top of that, using second round of malli-vqlidation, one could catch more, like integer min&max sizes, collection limits, closed maps, multis, sequences etc. The code would require access to:
1. function arguments
2. the :malli/schema value (from linter config). #2021-08-0911:19borkdudeand you would need to invoke malli itself as well right?#2021-08-0911:19ikitomminot sure if this is anyhow useful, but might be doable? At least emitting the malli-schema to the linter config would be a +1loc in malli.#2021-08-0911:19ikitommiyes#2021-08-0911:20borkdudein that case malli would have to be built into the clj-kondo or clojure-lsp binary, unless you run it on the JVM#2021-08-0911:21borkdudewhat you could do, as it is now, is programmatically generate hooks for each var that has a malli spec#2021-08-0911:21borkdudethere you can have access to the arguments and do whatever you like, throw exceptions. The hooks already have access to the linter config#2021-08-0911:22borkdudeand then you could write some validations like malli but in user space, just as a proof of concept#2021-08-0911:22borkdudeor you can fork clj-kondo and add malli to the type system and explore until you reach some interesting conclusions#2021-08-0911:22ikitommisounds fun#2021-08-0911:23borkdudeI'm also open to clj-kondo type system improvements, there are a few low hanging fruits perhaps#2021-08-0915:02ikitommi... also, if the config-file could be used as a.simple database, the tool could run Malli's check once for each var (gen-test) and mark :malli/check with the result (nil or error) - "function does not conform to it's schema, with arguments [0 -1], the result is -1, which is not a positive integer"#2021-08-0915:24Noah Bogartis it possible to enable generative testing (`{::m/function-checker mg/function-checker}`) for all function schemas during test runs without adding a (when (= :test (:env app)) ...) to every schema manually?#2021-08-0916:35ikitommithere is an issue to allow setting default options. Before that, you could redefine the var where it's read / defined?#2021-08-0916:35ikitommidirty, but works#2021-08-0916:35Noah Bogartyeah, maybe i’ll try that#2021-08-0916:35ikitommiAnd please write an issue#2021-08-0915:39richiardiandreaQuestion, say I want to try Malli but I don't have the fire power to rewrite all the specs...What would be the best "migration strategy" there?
We instrument every function in dev mode and tests and we use specs for our domain model and http param validation at the moment.
What I am worried about the former and the following - how can I instrument a function that has been partially covered with spec and partially with Malli?#2021-08-0916:37ikitommicreate a spec->malli converter, PR that and enjoy the ride? :face_with_cowboy_hat:#2021-08-0916:38ikitommiNot sure if the intstrumentatons stack, could just work?#2021-08-1003:34JoelI would think ':b' could be in square brackets: [:a {:x "s"} [:b]], but it doesn't work, i'd like to be able to nest :cat/n
(-> [:cat
[:enum :a]
[:map [:x string?]]
[:cat [:enum :b]]]
(mc/explain
[:a {:x "s"} :b])
(me/humanize))#2021-08-1004:44ikitommi@joel380 from the README:
As all these examples show, the "seqex" operators take any non-seqex child schema to mean a sequence of one element that matches that schema. To force that behaviour for a seqex child :schema can be used:
(m/validate
[:cat [:= :names] [:schema [:* string?]] [:= :nums] [:schema [:* number?]]]
[:names ["a" "b"] :nums [1 2 3]])
; => true
;; whereas
(m/validate
[:cat [:= :names] [:* string?] [:= :nums] [:* number?]]
[:names "a" "b" :nums 1 2 3])
; => true
#2021-08-1004:44ikitommie.g.
[:cat
[:enum :a]
[:map [:x string?]]
[:schema [:cat [:enum :b]]]]
#2021-08-1005:45ikitommi@emccue fixed in master. Also, you don’t need the malli.instrument/collect!, malli.dev/start! calls that.#2021-08-1005:45ikitommihttps://github.com/metosin/malli/pull/509#2021-08-1012:49emccueBut it won't call it on reload if i'm using the metadata form for the schema right?#2021-08-1013:11ikitommicurrently only on start!. I tried to add var-watching to the once-registered vars, but didn't have the skills to do that. start! & stop! should play nice with reloaded repl thou.#2021-08-1012:01Darnaroth DarnarowthQuestion, is it possible to add sci options on a custom decoder, or any ability to set sci options globally?#2021-08-1012:20Karol WójcikHi! How can I check if x satisfies the protocol y in malli metadata schema?#2021-08-1013:35Karol WójcikOk I found the answer!#2021-12-0907:52amithgeorgeI found this thread via Slack search. Please share the answer!!#2021-12-0914:28Karol WójcikCreate a schema from a -simple-schema
(def proto-schema (-simple-schema {:pred (fn [x] (satisfies? PROTO x)}))#2021-12-0914:41amithgeorgeThank you! Will try this. :thumbsup:#2021-08-1013:11ikitommicurrently only on start!. I tried to add var-watching to the once-registered vars, but didn't have the skills to do that. start! & stop! should play nice with reloaded repl thou.#2021-08-1013:39ikitommisilly-hacky-var-watching-diy-horror:
(defn plus1
"adds 1 to number"
{:malli/schma [:=> [:cat :int] :int]}
[x] (inc x))
(add-watch #'plus1 ::watch (fn [_ v _ _] (future (println v "=>" (-> v meta :malli/schema)))))
(defn plus1
"adds 1 to number"
{:malli/schema [:=> [:cat [:int {:min 0}]] :int]}
[x] (inc x))
; =prints=> #'user/plus1 => [:=> [:cat [:int {:min 0}]] :int]#2021-08-1013:39ikitommiif that could be made robust, could just re-define schmatized vars and the clj-kondo + instrumentation would follow instantly.#2021-08-1013:40ikitommiideas welcome (asked on #clojure too)#2021-08-1014:00emccuewell, if it needs to be "scheduled to fetch later" you could have something like#2021-08-1014:03emccue(defn schedule-to-refresh
[var]
(set! some-flag-another-thread-is-waiting-on true)
;; Or
(.push some-array-blocking-queue-another-thread-is-waiting-on ...))
(add-watch #'plus1
(fn [_ v _ _]
(schedule-to-refresh v)))#2021-08-1014:04emccueso at least you don't necessarily interact with the future threadpool#2021-08-1014:05emccueand it isn't a race condition#2021-08-1016:33robert-stuttaford@ikitommi thanks for the humanize buffs 🙂#2021-08-1100:08steveb8n@ikitommi I haven’t tested but would the new release be likely to improve this perf issue? https://gist.github.com/stevebuik/e63735d99fca94041120f9b0e25b616d#2021-08-1105:03Ben Slesstried it out of curiosity, looks like it was improved#2021-08-1108:19Karol WójcikWhat is wrong with this schema?
(def logger-pairs-schema
[:+ [:tuple keyword? string?]])
(defn pairs-conforms!
{:malli/schema [:=>
[:cat [logger-pairs-schema]]
nil?
;; [:or [nil? logger-pairs-schema]]
]}
[pairs]
)#2021-08-1108:20Karol Wójcik:malli.core/invalid-schema {:schema [:+ [:tuple #function[clojure.core/keyword?] #function[clojure.core/string?--5427]]]}
I don't understand why this throws an error#2021-08-1108:26Karol WójcikThis is thrown from the new reported
(ns dev
(:require
[malli.dev :as dev]
[malli.dev.pretty :as pretty]))
(dev/start! {:report (pretty/reporter)})#2021-08-1109:11Ben SlessTry not wrapping logger pairs schema in a vector?#2021-08-1110:07Karol WójcikI don't get it.
(m/validate logger-pairs-schema [[:x "something"]]) => true
@ikitommi isn't it a bug?#2021-08-1111:26ikitommino, just an extra vector:
(m/schema [:cat [[:+ [:tuple keyword? string?]]]])
; =throws=> :malli.core/invalid-schema
(m/schema [:cat [:+ [:tuple keyword? string?]]])
; => => [:cat [:+ [:tuple keyword? string?]]]
#2021-08-1111:27ikitommiwe could make the pretty printer capture schema-creation errors too in dev.#2021-08-1111:27ikitommisame result thou, just more fun with colors 🌈 !#2021-08-1112:20Karol WójcikAhh. Thank you @ikitommi :3#2021-08-1117:12emccue(mg/generate [:cat string? int?])
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :cat}#2021-08-1117:12emccueis there some context i'm missing?#2021-08-1117:12emccuecat doesn't seem to want to be a valid schema#2021-08-1117:20emccuenvm - seem to have had a really old version floating around the classpath#2021-08-1314:04emccueIt also just dawned on me that the metadata schemas mean I can publish a library with malli schemas and no dependency on malli#2021-08-1314:05emccuei have no doubt that was part of the intent and its silly I just realized it, but its still really cool#2021-08-1314:11Noah Bogartthat is cool!#2021-08-1410:02andre.richardsHi, :uuid schema does not produce correct humanized error:
(me/humanize (m/explain :uuid "foo"))
=> ["unknown error"]
uuid? produces this:
(me/humanize (m/explain uuid? "foo"))
=> ["should be a uuid"]
uuid? has an entry in malli.error but :uuid does not, so it looks like an easy fix.
Should I log a issue and raise a pull request?#2021-08-1410:20ikitommiPlease do!#2021-08-1410:40andre.richardsDone: https://github.com/metosin/malli/pull/512
Thanks for Malli!#2021-08-1410:51ikitommimerged, thanks!#2021-08-1411:39VladislavHi! It’s no option to assoc key with parameters (optional etc.) to schema hm for now, i’m right? I’m forced to use merge for that#2021-08-1413:50ikitommi@shishkov61 You can use a vector to assoc full entry, not at computer, but this should work:
(mu/assoc :map [:x {:optional true}] :int)#2021-08-1413:56Vladislavi’ll try it. thanks!#2021-08-1420:59Vladislav(m/validate
[:map [[:set string?] number?]]
{#{"a"} 1})
=> false
should it be like this?#2021-08-1421:08Vladislavactually, i don't see no docs about maps with keys which is not static values#2021-08-1421:11Vladislavshould i use :registry specified schemas for keys?:thinking_face:#2021-08-1421:15Vladislavno, guess i'd found it, but it works another way
https://github.com/metosin/malli#qualified-keys-in-a-map#2021-08-1421:28Vladislavo! map-of kinda work for that. but it will be complicated if there is no homogeneous keys needed#2021-08-1519:44Asko NōmmHi! I found a interesting issue in CLJS where Malli fails with "Invalid schema" when checking for object? , for example:
(m/validate [:or map? object?] input)
Fails. However whenever I remove the object?, it starts working. How do I validate a JS object with Malli if object? makes Malli fail?#2021-08-1520:07emccueonly certain predicates are "built in" - i.e. have an implicit mapping to something. map? gets swapped with [:map], effectively#2021-08-1520:08emccueif you want to use a custom predicate you can either use [:fn object?] - which won't have generation semantics if you use that for your project#2021-08-1520:10emccuethere is also a way to make a custom schema key/generator/etc. I just don't know it offhand#2021-08-1520:10emccuehttps://github.com/metosin/malli#custom-schema-types#2021-08-1520:10emccueit is documented here though#2021-08-1520:46Asko NōmmCheers @U3JH98J4R, this solved it perfectly. I didn't know that it swapped the predicates, this makes sense now!#2021-08-1608:19pithylessI'm working with the malli schema functions (the new dev error reporting is nice!) and I seem to be at a crossroads:
1. If I choose to use m/=> at the top-level:
a. I need to an explicit dependency on malli (not generally a problem with applications, but this is probably a no-go for libraries)
b. I cannot use non-standard registry schemas, since it is evaled at compile time.
c. I think even if I could force some preload logic to make sure my global malli registry is updated before other namespaces are loaded, this is both prone to errors and probably would still not work with external tools that work per-file (clj-kondo, etc)
2. If I choose to use :malli/schema metadata:
a. I am free to include it optionally without directly requiring malli
b. I can use non-standard registry schemas
c. The malli.dev/start! does not pick up var changes (need to force a refresh)
d. The metadata approach sometimes makes the function feel "verbose". This is a obviously a subjective feeling, but if the specs are verbose and the function body is short, sometimes the separate m/=> looks nicer.
Because of the problems with (1), I'm thinking of still focusing on (2). I can workaround (2c) and perhaps (2d) can be mitigated in certain cases with a (def foo-schema [:=> ...]) and then using (defn foo {:malli/schema #'foo-schema} ..) ?#2021-08-1608:19pithylessPosting these observations, in hopes that someone can set me straight on my misunderstanding; or perhaps suggest a better approach? :)#2021-08-1609:12ikitommi• 1b&c -> this should be resolved somehow, please write an issue
• 2c -> can be fixed (look up for var changes automatically)
• d -> true that
what might help if malli could pick up the function schemas from the actual functions behind the vars. it would allow one to say:
(defn kikka [])
;; just clojure
(defn => [v schema]
(alter-var-root v vary-meta assoc :malli/schema schema))
;; add a function schemas without deps to malli
(=> #'kikka [:=> :cat :any])
;; reading from function
(-> kikka meta :malli/schema) ; => [:=> :cat :any]
#2021-08-1609:40pithylessThat looks promising; I will try it out today and submit an issue later. Thanks!#2021-08-1609:12ikitommi• 1b&c -> this should be resolved somehow, please write an issue
• 2c -> can be fixed (look up for var changes automatically)
• d -> true that
what might help if malli could pick up the function schemas from the actual functions behind the vars. it would allow one to say:
(defn kikka [])
;; just clojure
(defn => [v schema]
(alter-var-root v vary-meta assoc :malli/schema schema))
;; add a function schemas without deps to malli
(=> #'kikka [:=> :cat :any])
;; reading from function
(-> kikka meta :malli/schema) ; => [:=> :cat :any]
#2021-08-1609:40Tiago Dall'OcaHello there#2021-08-1609:46Tiago Dall'OcaI'm trying to create a parser for converting malli's schemas to DTS files (typescript's type definitions file). One of the situations I'm running into is reading malli's schemas correctly, as they're plain data structures with semantic meaning, so I have to check for some options maps when parsing :map or even cases like https://github.com/metosin/malli#qualified-keys-in-a-map. I was thinking of writing a https://github.com/metosin/malli#parsing-values for helping me out with that, but maybe it already exists? Or is there an easier way?#2021-08-1609:59ikitommiSounds great, looking forward to this! You can as m/children and m/properties from a Schema instance. The first one returns the parsed enty-tuples [key properties value] if that helps.#2021-08-1612:53Tiago Dall'Ocam/children is exactly what I was looking for! Thanks :D#2021-08-1615:03richiardiandreaWill monitor this - we have TypeScript as frontend and indeed sharing the API specs would be super cool 🙂#2021-08-1617:22ikitommiBtw, the JSON Schema converter is a good example of generic transformer of malli->xyz, https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc#2022-08-0422:31richiardiandrea@U4U6BDQTE how far did you get with this project? Do you need/would you accept any help? 😄#2022-08-0422:31Tiago Dall'Ocahey there#2022-08-0422:32Tiago Dall'Ocaat the company I work for we started testing it in production actually#2022-08-0422:32Tiago Dall'Ocathe api is not stable#2022-08-0422:32Tiago Dall'Ocathe code is not pretty#2022-08-0422:32Tiago Dall'Ocabut it works for us haha#2022-08-0422:32Tiago Dall'Ocago check it out at flowyoumoney/malli-ts#2022-08-0422:33Tiago Dall'Ocawe didn't setup a collaboration structure just yet, but I think help is welcome!#2022-08-0422:33Tiago Dall'Oca@U0P7M2VHR#2022-08-0422:34Danny🙌#2022-08-0422:41richiardiandreanice that looks great! I think my company would greatly benefit from it and we'll try to give back 😄#2022-08-0423:08DannyAny idea yet how/where you would use it? 🙃#2021-08-1706:38ikitommiAnalysis about Malli Schema creation performance: https://github.com/metosin/malli/issues/513#2021-08-1708:24Ben Slesswoohoo, more performance#2021-08-1708:34Ben Slesscan you uplooad the raw svgs?#2021-08-1708:36ikitommicleared them up already, sorry.#2021-08-1708:48Ben Slessno biggy, I can repro#2021-08-1708:52Ben SlessBtw, regarding the memoization idea I floated
#?(:clj
(defn memoize!
[]
(doseq [v
[#'-validator
#'-explainer
#'-parser
#'-unparser
#'-transformer
#'-walk
#'-into-schema
#'-safe-pred]]
(alter-var-root v memoize))))
#?(:clj (defonce _memoized (memoize!)))
Is this sufficient?#2021-08-1714:14ikitommiunbounded cache? as Schemas don’t have custom equality defined, all instances are different. That would leak memory, a lot. But, interesting idea. Could use that in malli.dev to swap top-level functions into version that pretty print exceptions.#2021-08-1714:16ikitommiSafe thing might be to put the cache into options, so the user can control it. For schema instances, could be bolted into a registry?#2021-08-1714:18ikitommie.g. a registry that returns cached Schemas instead of IntoSchemas in cases it would benefit from caching, e.g. all immutable schmas like leaves without properties and children: :int could be cached, [:int {:title "wadawoksei, kuvavideo"}] is a bad candidate.#2021-08-1715:30Ben SlessWhat is the largest number of schemas you've seen defined?
I wonder if an unbounded cache will be good enough in most cases#2021-08-1717:58ikitommiinfinite. sending [:re #".*"] over the wire will always result in a new schema. Regexs don’t implement equality.
(= #".*" #".*") ; => false
#2021-08-1718:05Ben SlessWould it be worth implementing equality semantics for schemas?#2021-08-1718:06ikitommimaybe, there is malli.util/equals already, but it doesn’t take into account the possible different local registry bindings.#2021-08-1718:06ikitommiI recall there is an issue about pushing the local schema binding into “full”-form…#2021-08-1718:07ikitommiI think any option could effect how the schema works, so the equality might be heavy to calculate, might be wrong.#2021-08-1718:10ikitommibtw, thanks again for you efforts on perf, really appreciate it 🙇#2021-08-1718:10Ben SlessI'm just having lots of fun with it#2021-08-1718:10Ben SlessAnd it's all your fault for giving the perf talk at ClojureTRE 2019#2021-08-1718:11Ben Sless😄#2021-08-1816:03ikitommi#2021-08-1717:17Ben SlessIs it possible to use regex schema to say something like "I don't know what these two consecutive elements are but they should be identical"?, i.e. [1 1 1] would be valid but [1 2 1] would not?#2021-08-1723:59lsenjovSo all elements should be identical?#2021-08-1802:47lsenjovOr just certain positional elements?#2021-08-1804:50Ben SlessI gave a really bad example, sorry
Better example, for N=2 and T=int:
[1 1 7 7 8 8 2 2 3 3]#2021-08-1804:52ikitommidon't think so, :and would be the way to do this, but it currently pushes you out of the sequence.#2021-08-1804:56ikitommialso noticed that function schemas expect a :cat, which is bad as one can't wrap it like [:and [:catn [:min :int] [:max :int]] [:fn (fn [min max] (< min max))]]#2021-08-1805:43Ben SlessCould it be done with textual regular expressions using capture groups?#2021-08-1806:13Ben SlessSomething like (P)\1#2021-08-1905:51ikitommian inline/hidden :and would be my first guess, something like:
[:* [:and [:repeat {:min 2, :max 2} :int] [:fn (fn [[x y]] (= x y))]]]#2021-08-1905:52ikitommi:and is already weird, as the first thing is used in utilities. I guess this is the reason why Schema named it constrained. One Schema + constraints. Not “all the schmas”#2021-08-1913:08Ben Slesswas hoping to avoid fn#2021-08-1913:15ikitommiI think it’s the same thing as key-relations for maps, but… sequence relations instead.#2021-08-1913:17ikitommimalli + meander here could look like:
[:*
[:and
[:repeat {:min 2, :max 2} :int]
[:relations
'[?min ?max]
'[:= ?min ?max]]#2021-08-1913:19ikitommiand maps:
[:and
[:map
[:min :int]
[:max :int]]
[:relations
'{:min ?min, :max ?max}
'[:> ?min ?max]]]#2021-08-1913:34Ben Slesswith catn it's exactly the same#2021-08-1915:29kennyMalli often prefixes function names with -. Often this implies a private or internal function, yet lots of - functions are used and encouraged. What is the meaning behind the - prefix in the Malli codebase?#2021-08-1915:30emccue"read the docs"/"experts only"#2021-08-1915:34kennyI've read the docs several times and missed that section each time 🙂 Thanks. fwiw, those functions do not seem like "experts only." They have been critical to our use of Malli.#2021-08-2019:43dominicmIs there anything higher level than m/type for determining, e.g. sequences? I'd like to lump :* ,`:+`, :tuple, etc. together for my purposes when doing programmatic work on schemas. Same idea for pos-int? int?, etc.#2021-08-2019:58ikitommi@dominicm currently no, ideas welcome. There is an issue about derived types. int? is actually just a :`int`, silly to have to declare humanized error messages, generators, JSON schema mappings, transformers for both.#2021-08-2019:58ikitommihttps://github.com/metosin/malli/issues/264#2021-08-2020:02dominicm@ikitommi For sure. My use-case is along those lines. I'm trying to determine if a schema is, e.g. valid JSON (without transforms). Pairs nicely with Muuntaja 😉#2021-08-2020:03dominicmI'll have a look at the source of those features for my solution#2021-08-2104:01ikitommiyes, transformers know that, quick hack would be to loop the registered keys of json-transformer or even ask if there is anything to do:
(= identity
(m/decoder
schema
(mt/json-transformer))#2021-08-2110:50Ben SlessSome perf finding:
invoking map on arguments is the shortest path to .valAt without interop. Faster than get#2021-08-2114:24dominicm@ikitommi That doesn't quite work:
(m/decoder
[:map [{"type" "foo"} string?]]
(malli.transform/json-transformer))
Which can't be encoded as valid JSON.#2021-08-2114:26dominicmheh, jsonista converts the key to a edn string in that case: "{\"{\\\"type\\\" \\\"foo\\\"}\":\"bar\"}"#2021-08-2114:46ikitommi@dominicm right, there is no decoders registered for keys in json-transformer, it's expected to come clean from actual JSON decoder. And same for encoding, malli doesn't have JSON encoders for all schemas, as in the actual encoder (jsonista, Cheshire etc) already does that.#2021-08-2116:33Ben SlessOpened a bunch of small PRs relating to https://github.com/metosin/malli/issues/513 . I broke them down as much as possible to find the best performance gains for each change and to cover enough use cases in the benchmarks. Didn't want to end up accidentally slowing down one while speeding up the other (which I did, initially)#2021-08-2211:49anonimitorafHi guys, I'm looking to use malli for the following use cases...
Context:
• I'm working on a system that takes in data from various 3rd party sources (think scraping).
• The "scraped" data get transformed into an internal, universal data format that then get saved to DB
Problems to be solved:
• Validating that the 3rd party data are of some expected shape
Obvious how malli gets utilized here
• Mapping the 3rd party to the internal, universal data format
Is this possible to be done via custom transformers on each property of the scraped data?
I can give examples if needed#2021-08-2211:53Ben SlessDefinitely possible#2021-08-2211:53anonimitorafAwesome! I'll write up some dummy examples#2021-08-2211:53Ben Slessprobably more than one way for how you could do it, too#2021-08-2211:58anonimitorafActually, I'm finding it a bit hard to quickly write up dummy examples.
Do you mind linking docs on possible ways to do it @UK0810AQ2?#2021-08-2212:01anonimitorafActually here's an example:
(def source-shape
[:map
[:events [:sequential
[:map
[:id int?]
[:desc string?]
[:details-id int?]]]]
[:details [:sequential
[:map
[:id int?]
[:content string?]]]]])
Sample data:
{:events {:id 1
:desc "Blah"
:details-id 11}
:details [{:id 11
:content "Blargh"}]}
mapped to
[{:id 1
:description "Blah"
:details "Blargh"}]#2021-08-2212:08Ben SlessShould events be a sequence of maps?#2021-08-2212:15anonimitorafAh whoops yes. I'll correct my example#2021-08-2213:15Ben SlessVery roughly:
(defn index-by
[f xs]
(reduce (fn [acc x] (assoc acc (f x) x)) {} xs))
(defn join
[{:keys [events details]}]
(let [details (index-by :id details)]
(reduce
(fn [acc {:keys [details-id] :as event}]
(conj acc (-> event
(dissoc :details-id)
(assoc :details (get (get details details-id) :content)))))
[]
events)))
(def source-shape
[:map
{:decode/fun join}
[:events [:sequential
[:map
[:id int?]
[:desc string?]
[:details-id int?]]]]
[:details [:sequential
[:map
[:id int?]
[:content string?]]]]])
(def data
{:events [{:id 1
:desc "Blah"
:details-id 11}]
:details [{:id 11
:content "Blargh"}]})
(m/decode
source-shape
data
(mt/transformer {:name :fun}))
#2021-08-2213:15Ben Slessbut there should be a better day to do it#2021-08-2214:29pithylessIMO, things to be aware of:
1. decode is best-effort and will not throw errors if the validation does not pass
2. the transformation functions are expected to handle bad input without throwing errors
3. I think it's best to split the decode-validate-transform into several steps (each can have it's own validation)#2021-08-2214:30pithylessThis is how I'd approach it:
(def source-shape
[:map
[:events [:sequential
[:map
[:id int?]
[:desc string?]
[:details-id int?]]]]
[:details [:sequential
[:map
[:id int?]
[:content string?]]]]])
(def destination-shape
[:sequential
[:map
[:id int?]
[:description string?]
[:details string?]]])#2021-08-2214:30pithyless(def sample-data
{:events [{:id "1"
:desc "Blah"
:details-id "11"}
{:id "2"
:desc "Blah 2"
:details-id "12"}]
:details [{:id "11"
:content "Blargh"}
{:id "12"
:content "Blargh 2"}]})
;; Example broken input
(def sample-data2
{:events [{:id nil
:desc "Blah"
:details-id 11}]
:details [{:id "11"
:content "Blargh"}]})#2021-08-2214:30pithyless(let [raw-data sample-data
;; First, normalize input (this is best-effort and does not throw errors).
;; This can take care of things like converting strings to integers, because
;; the serialization protocol could not represent integers, etc.
decoded (m/decode source-shape raw-data mt/string-transformer)
;; Second, validate that the normalized input matches our expectations.
;; NOTE: you can use m/validate instead of m/parse if schema has branching logic,
;; but you don't want to take advantage of the extra information at this point.
;; NOTE 2: you could use a different, more strict schema at this point.
parsed (m/parse source-shape decoded)
_ (when (identical? parsed ::m/invalid)
(throw (ex-info "Invalid Input" (m/explain source-shape decoded))))
;; At this point we've validated input and want to do transformation
transformed (custom-transform-logic parsed)
;; And would be good idea to validate transformation worked...
_ (when-not (m/validate destination-shape transformed)
(throw (ex-info "Invalid Transformation" (m/explain destination-shape transformed))))]
transformed)#2021-08-2214:32pithylessThe custom-transform-logic could be the approach @UK0810AQ2 mentioned; but if we're doing these kind of map/join transformations, it might be worth it to checkout meander (especially when the cases become more complicated to grok):
(defn custom-transform-logic
[data]
(-> data
(meander/search
{:events (meander/scan
{:id ?event-id
:desc ?description
:details-id ?details-id})
:details (meander/scan
{:id ?details-id
:content ?details})}
{:id ?event-id
:description ?description
:details ?details})
vec))#2021-08-2214:32pithyless^ FYI @UR37CBF8D :)#2021-08-2214:33pithylessAlso important to mention I used m/validate m/parse etc., but these should all be replaced with m/validator, m/parser etc. in production code#2021-08-2214:37pithylessI apologize for pasting so much code into slack; here's a gist: https://gist.github.com/pithyless/0be222e1b1b3bca0239a9ca07d1b34c2#2021-08-2217:17schmeeI second the recommendation for Meander, perfect use-case for it: https://github.com/noprompt/meander#2021-08-2223:11anonimitorafThanks for the replies guys! I'll check them out after work 🙂#2021-08-2313:20ikitommiYes, meander is the king in complex transformations and works nicely with malli. Great example @U05476190!#2021-08-2314:39pithylessThanks @U055NJ5CC, I was hoping someone more experienced would confirm if the approach was reasonable.#2021-08-2412:58anonimitorafHey guys, again, thanks for these! I somewhat understand how they work.
Follow up question:
Is there a way to do it via metadata on the malli's schema fields and I somehow manipulate those metadata?
I'm thinking something like
(def source-shape
[:map
[:events [:sequential
[:map
[:id int?]
[:desc
{:custom/map-to :description} ;; <----- like this
string?]
[:details-id int?]]]]
[:details [:sequential
[:map
[:id int?]
[:content string?]]]]])
?#2021-08-2218:09Ben SlessWrote a mutable entries parser just to see if it was possible.
Eliminated almost all the overhead related to parsing entries.
It's a disgustingly ugly direct translation of the code, but it works and cut down about 1us / entry#2021-08-2218:43Ben SlessBehold, my horrible creation#2021-08-2218:43Ben Slesshttps://gist.github.com/bsless/c21995e5702c2fa6d954312786da197b#2021-08-2219:43ikitommi:smiling_face_with_3_hearts:#2021-08-2219:43ikitommimy current project use case is on cljs-side btw.#2021-08-2219:47Ben Sless😄#2021-08-2219:49Ben SlessI'm just surprised it works
For cljs we can do what I started with which was propagating the bindings for every case, it should save some, too#2021-08-2219:17respatializedis there a more idiomatic way of testing two schemas for equality than comparing them with malli.util/to-map-syntax?#2021-08-2219:18respatializedlol, just noticed malli.util/equals, please disregard#2021-08-2219:27Ben SlessCaveat emptor
(mu/equals (m/schema [:re #"hi"]) (m/schema [:re #"hi"]))
false
#2021-08-2219:41ikitommiregexp schemas should be thrown away, could be just a property in :string schema#2021-08-2219:42ikitommi[:string {:pattern "hi"}]#2021-08-2316:59pithylessI'm seeing some weird behavior that may (or may not) be related to the intersection of using defprotocol m/=> and malli.dev/start! where I'm getting a StackOverflow exception when running an instrumented function. If I comment out the m/=> everything works fine. Before I go down a rabbit hole of trying to come up with a minimal repro, does this sound familiar to someone?#2021-08-2321:38winsome@mrdalloca, I see up above that you're exploring using malli to generate typescript definitions! I'm just starting to look into that as well, are you open to sharing what you've learned so far? I'm happy to help#2021-08-2321:42Tiago Dall'Ocaheyy#2021-08-2321:42Tiago Dall'Ocasure! I've progressed quite a bit actually#2021-08-2321:43Tiago Dall'OcaI'm working with a draft of it by now#2021-08-2321:43Tiago Dall'Ocaso I'm not happy with the code organization and naming in general#2021-08-2321:43Tiago Dall'Ocavery early stages#2021-08-2321:43Tiago Dall'OcaBUT#2021-08-2321:43Tiago Dall'OcaI'm already generating type definitions#2021-08-2321:44winsomeFantastic! I've got to hop off for other life things in a minute, but do you mind if I message you later? I'd love to try the basics#2021-08-2321:51Tiago Dall'Oca#2021-08-2321:53Tiago Dall'OcaI don't have time left for today, but we can go into details about this tomorrow :))#2021-08-2321:54Tiago Dall'OcaI wake up at 09:00 UTC#2021-08-2403:31datranI won't be about until 14:00 UTC, but I'll ping you then :thumbsup:#2021-08-2323:00respatializedI'm hitting a wall trying to write a schema walker that recursively transforms :altn to :alt, :catn to :cat, etc. The implementation is straightforward for simple schemas, but I encounter failures when I use :refs and try to transform the registry along with the values - the registry doesn't get passed down when the walker descends into the :ref schemas.
I'm not sure about the best way to proceed. Rebinding mr/*registry* with a dynamic registry that contains the top-level schema registry seems expedient, but I don't really understand the registry code that well yet which seems perilous.#2021-08-2323:02respatializedHere's what I've got so far. It fails on the call to mu/to-map-syntax in the inner forms because of an invalid :ref error.#2021-08-2414:03AkizHi, do you know what is happening here?
bq2pg.config> (m/validator [:map [:export? boolean?] [:import? boolean?] [:gcs-name string?]])
#function[malli.core/-map-schema/reify/reify--9403/fn--9419]
But using > 3 elements fails…
bq2pg.config> (m/validator [:map [:export? boolean?] [:import? boolean?] [:gcs-name string?] [:extra string?]])
Execution error (NoClassDefFoundError) at malli.impl.util/f (util.cljc:54).
malli/impl/util$f__7020__7021$fn__7045
#2021-08-2414:45Akiz@zikajk This happens only when I use
malli.clj-kondo +
(-> (mc/collect *ns*) (mc/linter-config))
(mc/emit!)
(md/start!)
after some time… I can not reproduce it well enough 😞#2021-08-2421:07niclasnilssonI’m trying out the instrumentation, and I’m probably doing something wrong, but I just can’t see it right now. The simple cases with primitives from the docs works, but when I try to validate a more “normal” schema, I can’t get it to work. This is an example:
(def coord-schema
[:map
[:x :int]
[:y :int]])
(defn down
{:malli/schema [:=> [:cat coord-schema] [coord-schema]]}
[coord]
(update coord :y dec))
When I then do (dev/start!) I get the error
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:119).
:malli.core/invalid-schema {:schema [:map [:x :int] [:y :int]]}
What am I missing?#2021-08-2421:55niclasnilssonLooks like I was missing [:schema …]. This works.
(defn down
"Some docstring"
{:malli/schema [:=> [:cat [:schema coord-schema]] [:schema coord-schema]]}
[coord]
(update coord :y dec))
#2021-08-2506:19AkizDo you have to use schema when you do not use instrumentation?#2021-08-2506:38ikitommi@U48DE3SHM it should be:
(defn down
{:malli/schema [:=> [:cat coord-schema] coord-schema]}
[coord]
(update coord :y dec))
e.g. don’t wrap schema in empty vector, invalid malli syntax.#2021-08-2510:41niclasnilssonAh, so my initial problem was the vector on the output all along. I see. Thanks, @U055NJ5CC.#2021-08-2510:43niclasnilsson@zikajk, I don’t know, but my goal was to use the devevelopment instrumentation, using (dev/start!)#2021-08-2510:59ikitommithe error message was really bad btw: `
malli.core/invalid-schema {:schema [:map [:x :int] [:y :int]]}
could have been something like:
malli.core/invalid-schema {:type [:map [:x :int] [:y :int]]
:schema [[:map [:x :int] [:y :int]]]}
#2021-08-2511:51Akiz@U48DE3SHM I experienced two different errors during schema validation when using instrumentation. Restarting helped though…#2021-08-2605:49niclasnilsson@zikajk, ah, thanks, I’ll keep that in mind going forward.#2021-08-2506:40ikitommi@zikajk looks weird. md/start! btw calls the clj-kondo too, so just md/start! and you are done#2021-08-2506:40ikitommiif you can repro, I can take a look#2021-08-2507:36Akiz@ikitommi I will try, I experienced this after long running repl session so there could be a lot of reasons.
After restarting - everything is working great.#2021-08-2506:42ikitommiworks here:
(def valid?
(m/validator
[:map
[:export? boolean?]
[:import? boolean?]
[:gcs-name string?]
[:extra string?]]))
(valid?
{:export? true
:import? true
:gcs-name "kikka"
:extra "foo"})
; => true#2021-08-2506:55Karol WójcikI'm trying to use reitit-malli coercion with malli 0.6.1. When evaling
[reitit.coercion.malli :as rss] I'm getting
------ WARNING - :fn-arity -----------------------------------------------------
Resource: <eval>:2007:43
Wrong number of args (3) passed to malli.core/-fail!
--------------------------------------------------------------------------------
Can someone reproduce?#2021-08-2507:15ikitommi@karol.wojcik that’s a bug, the 3-arity was removed. please write an issue, need to fix reitit (or revert the change if it’s really needed)#2021-08-2507:30Karol WójcikDone: https://github.com/metosin/reitit/issues/504#2021-08-2507:27Karol WójcikThank you @ikitommi for blazing fast response. I will write an issue! No worries I can wait 🙂#2021-08-2508:16ikitommiEveryone: malli master has massive performance improvements related to Schema creation and transformation performance. If you are using malli especially with cljs, you might see order of magnitude improvements to startup-times. Please try and report both gains and bugs. Big thanks for @ben.sless for much of the hard work.
(def ?schema
[:map
[:x boolean?]
[:y {:optional true} int?]
[:z [:map
[:x boolean?]
[:y {:optional true} int?]]]])
;; 44µs => 8.5µs (5x)
(def schema (m/schema ?schema))
;; 26µs => 1.3µs (20x)
(m/walk schema (m/schema-walker identity))
;; 51µs => 6.5µs (8x)
(mu/closed-schema schema)
… there is still a lot of room for improvements, but would like to release these soon. Issue tracking, alternative ways to build large schema systems described and comment welcome on https://github.com/metosin/malli/issues/513.#2021-08-2510:20Ben Sless@ikitommi Here's a riddle I can't find a good answer to, how come this is significantly faster than a PersistentHashMap backed registry
(defn fast-registry
[m]
(let [pred (comp keyword? key)
im (doto (new IdentityHashMap 1000) (.putAll (into {} (filter pred) m)))
-m (doto (new HashMap 1000 0.25) (.putAll (into {} (remove pred) m)))]
(reify mr/Registry
(-schema [_ k]
(let [^Map m (if (instance? Keyword k) im (if (instance? Var k) im -m))]
(.get ^Map m k)))
(-schemas [_] m))))
By about 35 ns for keyword lookup
compare
(cc/quick-bench (m/schema :int {:registry r}))
(cc/quick-bench (m/schema :int))
But the registry lookup isn't that much faster, only ~7 ns
(cc/quick-bench (mr/-schema r :int))
(cc/quick-bench (mr/-schema m/default-registry :int))
#2021-08-2510:50ikitommi🤷 micro benchmarks are hard. I would ask JMH how they really differ. Just swapping order of the tests, you might get different results. Also, more code, JIT behaves differently.#2021-08-2510:51ikitommiSuper quick googling: https://www.oracle.com/technical-resources/articles/java/architect-benchmarking.html#2021-08-2512:12Ben SlessInteresting, JMH gives opposite results#2021-08-2511:03lreadThe malli readme might be a good candidate for https://github.com/lread/test-doc-blocks. I could take a whirl at it if there is any interest.#2021-08-2520:43lread@ikitommi I won’t proceed unless you indicate an interest. Might be your cup of tea and might not! Happy to do the PR for you to evaluate then decide if you like - or not.#2021-08-2511:55martinklepschI’m getting an fn-arity warning when requiring malli.core in a cljs repl:
; eval (current-form): (require 'malli.core)
; (err) ------ WARNING - :fn-arity -----------------------------------------------------
; (err) Resource: <eval>:2007:43
; (err) Wrong number of args (3) passed to malli.core/-fail!
; (err) --------------------------------------------------------------------------------
#2021-08-2516:42Karol WójcikHad the same issue. It's fixed in master#2021-08-2517:56martinklepschthanks Karol!#2021-08-2520:03Karol WójcikYou welcome Martin! 🙂#2021-08-2613:34martinklepsch@ikitommi do you think it’s worth documenting this in the changelog for 0.7?#2021-08-2614:13ikitommiPR welcome @U050TNB9F, but there is a a already this line:
> * fixed arity error in `m/function-schema`#2021-08-2614:13ikitommidoes not pinpoint the problem very well :thinking_face:#2021-08-2512:37AkizHi, I am getting
Expected: regular expression, received: string.
in clj-kondo lsp / Emacs combo with instrumentation
for :
(def gcs-uri "")
(gcs/get-blob-moddate gcs-uri)
and these are my schemas:
(m/=> get-blob-moddate [:=>
[:cat Gcs-uri]
[:or
[:fn #(= (type %) org.joda.time.DateTime)]
nil?]])
(def Gcs-uri
[:re #"gs:\/\/.*"])#2021-08-2516:42Karol WójcikIs it possible to instrument function via metadata in Clojurescript?#2021-08-2517:09ikitommisomeone needs to port the tooling to support cljs. I suck at macro+interop, so not going to do that any time soon. help most welcome.#2021-08-2517:10Karol WójcikI will take a look into it when I have some time 🙂#2021-08-2517:22lreadAm having a great time experimenting with malli to validate cljdoc’s cljdoc.edn . The docs and demos are great in helping me to understand how things work. Thank you for all that effort. As a newb, I was wondering about the following:
1. The https://malli.io/?value=%5B1%20%5B2%20%5B3%20%5B4%20nil%5D%5D%5D%5D&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22ConsCell%22%20%5B%3Amaybe%20%5B%3Atuple%20%3Aint%20%5B%3Aref%20%22ConsCell%22%5D%5D%5D%7D%7D%0A%20%22ConsCell%22%5D uses [:ref "ConsCell"] but the https://malli.io/?value=%5B%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D%5D&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D uses [:schema [:ref "hiccup"]]. Are [:ref x] and [:schema [:ref x]] equivalent?
2. I was unfamiliar with :maybe and was not confident what it did. From the examples it seems that [:maybe x] validates for x or nil?
3. I was unfamiliar with :tuple and now think I understand it just fine. But maybe a sentence contrasting it with :vector (if that makes sense) would be helpful to newcomers.
4. Very subjective, but I would put the motivation section before examples. As a newbie, this kind of info would be helpful to me up front.
If updates to docs and demos for any the above would be helpful I am happy to make those wee changes.
Also once I’ve finalized my cljdoc.edn validation I would be happy to add that as a, perhaps familiar, example to docs, demos or both.#2021-08-2517:46Ben SlessI think schema of ref cannot be derefed. This lets you write things like recursive sequence schemas#2021-08-2518:08lreadThanks @ben.sless! - Are you suggesting maybe the hiccup demo might be a bit off?#2021-08-2518:09Ben SlessCan't say without giving it a deeper look. I bumped by head against it separately trying to port the datalog BNF to malli#2021-08-2518:10Ben SlessSchema schemas create a "hard" boundary when schemas are compiled and you don't get deref-ed recursively ad inifinitum#2021-08-2518:10Ben Slessyour understanding of :maybe is correct#2021-08-2518:11Ben Slesswith touple, just making sure, it is a heterogeneous fixed length product type#2021-08-2518:12lreadis a tuple necessarily a vector?#2021-08-2518:12Ben SlessIt seems so#2021-08-2518:13lreadCoolio, so I think I get it and could maybe add a sentence in the README.#2021-08-2518:13Ben SlessI'm working on a getting started guide 🙃#2021-08-2518:14lreadOh cool! Maybe I don’t need to update README then.#2021-08-2518:14Ben SlessIt's a good idea to do so anyway, it's a small change you can do now, the guide will take some time#2021-08-2518:14lreadPing me if you want a reviewer. I’m a newb!#2021-08-2518:15Ben SlessI have a bunch of guinea pigs at work 😛#2021-08-2518:16Ben SlessBut once it's sorta baked I'll go hunting for external perspectives#2021-08-2518:16Ben Slessso expect a ping in your future!#2021-08-2518:16Ben SlessJust, you know, don't deref it or anything#2021-08-2518:19lreadWhen working on your datalog BNF were you able to generate and sample without sporadic StackOverflowError exceptions?#2021-08-2518:20Ben SlessYes, let me check if it's public I'll send it#2021-08-2518:22lreadtx#2021-08-2518:23Ben Slesshere you go https://github.com/bsless/mallilog/blob/master/src/mallilog/impl/schema.clj#2021-08-2518:24Ben SlessNote the abundance of [:schema [:ref ,,,]]#2021-08-2518:26lreadYeah! So maybe that tells us something about my first question(?). You don’t seem to use :ref without it wrapped in :schema.#2021-08-2518:28Ben SlessThat's because my schemas are mostly recursive but I probably don't need to do it everywhere#2021-08-2518:29Ben SlessI can try by process of elimination to knock it out and see what's up but it's pedantic and didn't really feel like fighting with it#2021-08-2518:29Ben Slessbesides, I consider it a bit of an anti-feature#2021-08-2518:43lreadthe having to do something special for the recursive case?#2021-08-2518:43Ben SlessYeah, I'd expect a ref to be enough#2021-08-3119:47lreadTook a stab at https://github.com/metosin/malli/pull/537.#2021-08-2521:22lreadHiya mallians! I’m making some good progress with my cljdoc.edn malli schema. I am, though, getting some StackOverflowError exceptions when generating samples. This might be normal, I’m new, what do I know? simple_smile
Rather than share and diagnose my cljdoc.edn schema, it might be more fruitful to work from the familiar.
The Hiccup schema from the malli README seems to have-ish the pattern of recursion I want for cljdoc.edn.
If I take the Hiccup schema unchanged from the README I do not seem to suffer StackOverflowError exceptions.
(def Hiccup
[:schema {:registry {"hiccup" [:orn
[:node [:catn
[:name keyword?]
[:props [:? [:map-of keyword? any?]]]
[:children [:* [:schema [:ref "hiccup"]]]]]]
[:primitive [:orn
[:nil nil?]
[:boolean boolean?]
[:number number?]
[:text string?]]]]}}
"hiccup"])
(mg/sample Hiccup)
But if I yank out some stuff and make the schema look a little more like my cljdoc.edn schema usage, I can easily trigger StackOverflowError exceptions by re-evaluating (mg/sample Hiccup2) a handful of times:
(def Hiccup2
[:schema {:registry {"hiccup" [:catn
[:name keyword?]
[:props [:? [:map-of keyword? any?]]]
[:children [:* [:schema [:ref "hiccup"]]]]]}}
"hiccup"])
(mg/sample Hiccup2)
I am likely quite naive in my understanding here. I’m guessing if I somehow limited the maximum tree depth (which would be reasonable thing to do anyway for my use case) that would help but I am not sure how to express that.
I am using malli 0.6.1.#2021-08-2611:15jeroenvandijk@UE21H2HHD Without saying i’m an authority here, it seems that your schema doesn’t have a stop condition? So endless recursion? Whereas the hiccup one has an alternative which stops the recursion. Makes sense?#2021-08-2611:15jeroenvandijkThe only stop condition you have is when there are 0 children for one node#2021-08-2614:42lreadAh right… thanks @U0FT7SRLP! That makes perfect sense. For the generator, the :orn is probably a 50/50 coin toss for termination whereas hitting 0 children for termination is much less likely.#2021-08-2614:46jeroenvandijkYeah and then there is the branching of the children 😅 1 child still ok, but 2 or more and there you go#2021-08-2614:48lreadSo constructing a schema for validation is one thing and then adapting it to to support sensible generation is another consideration.#2021-08-2614:49lreadOr maybe folks write custom generators?#2021-08-2615:00lreadOh I see :gen/* stuff can influence generation, I’ll poke around more.#2021-08-2616:02jeroenvandijkWith clojure.spec https://clojure.org/guides/spec#_custom_generators, so I’m guessing this would make sense for Malli too. (I haven’t done it yet with Malli)#2021-08-2712:03martinklepschHow would I express that a schema can only have key :a OR key :b but not both? #2021-08-2713:09lreadWould something like this work?:
[:and
[:map
[:x {:optional true} int?]
[:y {:optional true} int?]]
[:fn {:error/message "Only one of :x or :y allowed"} (fn [{:keys [x y]}] (not (and x y)))]]#2021-08-2713:14lreadIn https://malli.io/?value=%7B%3Ax%201%20%3Ay%202%7D&schema=%20%5B%3Aand%0A%20%20%20%5B%3Amap%0A%20%20%20%20%5B%3Ax%20%7B%3Aoptional%20true%7D%20int%3F%5D%0A%20%20%20%20%5B%3Ay%20%7B%3Aoptional%20true%7D%20int%3F%5D%5D%0A%20%20%20%5B%3Afn%20%7B%3Aerror%2Fmessage%20%22Only%20one%20of%20%3Ax%20or%20%3Ay%20allowed%22%7D%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(not%20(and%20x%20y)))%5D%5D#2021-08-2713:59ikitommithere are ideas around declarative way of doing that in the future, either using datascript or meander syntax.#2021-08-2713:31mike_ananevHello! I'm trying to use an example from malli docs. Why I got an error?
(defn plus1 [x] (inc x))
(m/=> plus1 [:=> [:cat [:int {:max 5}]] [:int {:max 6}]])
;; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:119). :malli.core/invalid-schema {:schema :=>}#2021-08-2913:48mike_ananevhmm, I see a strange behaviour. If I run in terminal repl, everything woks fine. If I run via Idea+Cursive then I see this error. Btw, I deleted .cpcache completely.#2021-08-2913:50mike_ananevIn Emacs I see the same error#2021-08-2914:20mike_ananev@U055NJ5CC I've found the reason of that error.
If I add `
:jvm-opts ["-Dmalli.registry/type=custom"]
to deps.end alias, then I see this behaviour. Is it normal?#2021-08-2916:17mike_ananevfound solution for this: `
(mr/set-default-registry! (m/default-schemas))
#2021-08-2916:58ikitommigood to hear you got it reaolved#2021-08-2713:31mike_ananevmalli version 0.6.1#2021-08-2714:16Brett RowberryI think the answer to this question is probably going to be "just use reitit".
Anyway, compojure.api has support for coercion and validation via Schema and spec. Are there examples on how to use malli instead?#2021-08-2714:22Tiago Dall'OcaMALLI->TS - a sneak peek#2021-08-2806:38anonimitorafHi guys (kinda new to malli), when walking the schema, how would I walk the schema and transform the value of each node with custom logic?
I've looked at the src of the json-transformer, etc but they go over my head at the moment 😞
e.g.
(def my-schema [:map
[:a [string? {:custom/upper}]]
[:b [int? {:custom/square}]]])
(def my-data {:a "a", :b 2})
;; Transform, key :a to be upper-cased and :b squared
{:a "a", :b 5} => {:a "A", :b 25} #2021-08-2808:59pithylessBut that's just my opinion; perhaps other #malli users will weigh-in with their own perspectives :)#2021-08-3115:13olynice I came to ask pretty much this, I have a malli spec which takes keywords and goes through the default json transformer and failes because its a string, looks like I can just add :decode/json {:leave keyword} to the spec and it should transform to the correct format#2021-08-2808:16pithylessHmmm, the default interceptor transformers (which I admit I have not tried to customize) do not behave as I would have expected:
(m/decode [int? {:decode/json {:leave 'inc}}] 5 (mt/json-transformer))
;; => 6
(m/decode [int? {:decode/json {:leave 'inc}}] "5" (mt/json-transformer))
;; (expected) error! int is a valid json value, so should not have been represented as string
(m/decode [int? {:decode/string {:leave 'inc}}] 5 (mt/string-transformer))
;; => 6
(m/decode [int? {:decode/string {:leave 'inc}}] "5" (mt/string-transformer))
;; (unexpected) error! I would have expected the default string enter/leave interceptors to run by now, so we can `(inc 5)`#2021-08-2808:59pithylessBut that's just my opinion; perhaps other #malli users will weigh-in with their own perspectives :)#2021-08-3008:36andre.richardsIs there a way to pass custom registry to function schemas (metadata function schemas specifically)?
I.e. when calling m/validate you can specify :registry key in options map, but not sure how to do this for functions. I tried to add :registry to options map passed to m.dev/start! but that seems to be ignored.#2021-09-0114:53CélioIs there a way to customize the “missing required key” error message? For example, the code below returns the default missing key error message instead of something's wrong which makes sense because :error/message relates to the contents of :something, not to whether the key is present. It would be great if we could customize missing key errors.
(me/humanize (m/explain
[:map
[:something
[:string {:error/message "something's wrong"}]]]
{}))#2021-09-0115:16dergutemoritzHave you tried putting it on the entry's props?#2021-09-0115:16dergutemoritzas in:
[:something {:error/message "something's wrong"} :string]#2021-09-0217:01Célio@U06GVE6NR Yes, the result is the same.#2021-09-0118:06kirkedWhat's the best way to specify a schema for a map with both fixed keys and dynamic keys? Conceptually what I'd like is something like this schema to specify that a step has a :run keyword and some dynamic keys, and a workflow has :name and :start keywords and some dynamic keys:
(def step
[:and [:map [:run :qualified-keyword]]
[:map-of :keyword [:map [:next :keyword]]]])
(def workflow
[:and [:map [:name :string]
[:start :keyword]]
[:map-of :keyword step]])
The data has this shape:
{:name "my-workflow"
:start :begin
:begin
{:run :app/init
:ok {:next :do-work}
:fail {:next :done}}
:do-work
{:run :app/process-queue
:ok {:next :done}}
:done
{:run :app/cleanup}}}
The problem I've run into is that I need to #(apply dissoc % fixed-keys) before the :map-of schema is evaluated for any map matching this pattern.
Maybe there's a better way?#2021-09-0817:20ikitommisorry, no, but there is an issue for that: https://github.com/metosin/malli/issues/43 @U09EGLJLB#2021-09-0211:21borkdude@ikitommi Congrats on the CT funding!#2021-09-0213:11ikitommithanks! and congrats on you @borkdude too on the full year! (the application on the page is the old one, those are all done, send email to update that)#2021-09-0604:59emccueremembering typescript has https://github.com/DefinitelyTyped/DefinitelyTyped and wondering if it would be sensible to do something similar for clj* libraries and malli#2021-09-0605:00emccueMotivating example for me most recently is datomic's api - it gives annoying and obtuse errors if you pass bad data and spec'ing inputs at dev time would be helpful#2021-09-0605:01emccue(alter-meta!
#'d/transact
assoc
:malli/schema
[:=>
[:cat some? [:map [:tx-data some?]]]
[:map
[:db-before any?]
[:db-after any?]
[:tx-data any?]
[:tempids any?]]])#2021-09-0605:03emccueidk what the best approach is - maybe artificially associate with a namespace at first instead of the metadata?#2021-09-0605:05emccue(defn register-possibly-nonexistant-schema
[var-path schema]
(m/-register-function-schema! (symbol (namespace var-path))
(symbol (name var-path))
schema
nil))
(register-possibly-nonexistant-schema
'datomic.client.api/transact
[:=>
[:cat some? [:map [:tx-data some?]]]
[:map
[:db-before any?]
[:db-after any?]
[:tx-data any?]
[:tempids any?]]])#2021-09-0605:05emccueor maybe just ignore it and let library authors do it themselves#2021-09-0605:05emccueand fight the inevitable war with whatever is finally christened spec 2#2021-09-0605:13emccueor maybe the library has its own registry you can select from and merge into your own / the global registry#2021-09-0605:13emccueidk#2021-09-0605:17lsenjovPick a top level namespaces that does all the (m/=> ...) declarations, and they can include it themselves if they feel so inclined?#2021-09-0617:17Noah BogartClj-kondo has this for linting. Maybe malli could adopt a similar pattern? #2021-09-0617:31ikitommithat would be interesting. One of my goals with clojurists-together is to get a robust schema inferrer: for runtime for inferring from schemas from samples, but could be used to pull out schemas from tests (annotate functions in dev-mode to infer input & output) schemas. Not a silver bullet, but would be “for free”. Manually writing functios schmas is always better, but also, a lot of work.#2021-09-0812:02aaron51Is there a malli helper that converts a map to a := schema?
Input is {:a 1 :b 2}
Output is [:map [:a [:= 1]] [:b [:= 2]]]
I’m trying to validate a subset of a map (and the input is the sub map)#2021-09-0812:06ikitommi@aaron51 not yet. but plan is to build both a first-class inferrer and re-visit the type hierarchies so we could narrow down things. before that, you could hack the malli.provider, it’s quite small and relatively simple.#2021-09-0812:23aaron51Understood, thank you 👍#2021-09-1211:07Ben SlessThis is somewhere between malli and #reitit, is there a way to know the schema with which validation passed in cases of or and multi?#2021-09-1214:53respatializedI asked this question a while back; there's not a good way to do this without orn. If you know all the validation paths at compile time they can be enumerated. But the use case I was thinking about, of sum types that are created at runtime, was still tricky to match on. @U055NJ5CC suggested a simple function for converting the or to orn with numbers as the keys so that I could know which was matched.
Not sure if anything has changed in the library since I asked about it, so this solution may be out of date.#2021-09-1214:57Ben SlessEnded up going with orn and parse in the end
I wonder how it interacts with decoding. Are conflicts possible?#2021-09-1218:11rutledgepaulvI've used a custom decoder for this that attaches the schema to the value as metadata#2021-09-1620:13Yehonathan SharvitAre there some tools out there that generate a data model diagram (in a format like DOT or PlantUML) from a JSON Schema?#2021-09-1712:10emccuemalli can make uml from its own schema#2021-09-1820:32Yehonathan SharvitI know. But my question is about JSON schema#2021-09-1820:32Yehonathan SharvitIs there a way to transform a JSON schema into a Malli schema?#2021-09-1820:58emccueMaybe an equivalent question is "are all things representable in JSON schema representable in malli"?#2021-09-1820:59emccueBecause i don't think anyone has done that transformation yet#2021-09-1821:10nivekuilcould you infer a malli schema from the corresponding json object?#2021-09-1916:24ikitommiI noticed you got already some answer on Twitter @U0L91U7A8, could you share those here too?#2021-09-1917:29Yehonathan SharvitUnfortunately, none of the answers were satisfactory.#2021-09-1917:30Yehonathan SharvitThe best I could do is to manually convert the JSON schema into Malli and use Malli to generate a plantuml diagram#2021-09-1917:32ikitommiOk. The new map-syntax is quite close to JSON Schema, which should make the JSON Schema -> Malli streightworward.#2021-09-2208:29Yehonathan SharvitIndeed.
@U055NJ5CC What was the motivation for adding the new map-syntax?#2021-09-2210:24ikitommi1. order of magnitude faster way to instantiate schemas (e.g. ast IS the parsed results, no need to parse hiccup) -> matters on js & mobile
2. the old map-syntax was too generic to be useful
3. creating schemas programmatically behind the scenes is easier with maps (e.g. when inferring schemas from data)
4. it’s good to have options 😎#2021-09-2215:01Yehonathan SharvitMakes sense. Thanks#2021-09-1820:32Yehonathan SharvitIs there a way to transform a JSON schema into a Malli schema?#2021-09-2210:24ikitommi1. order of magnitude faster way to instantiate schemas (e.g. ast IS the parsed results, no need to parse hiccup) -> matters on js & mobile
2. the old map-syntax was too generic to be useful
3. creating schemas programmatically behind the scenes is easier with maps (e.g. when inferring schemas from data)
4. it’s good to have options 😎#2021-09-1914:30WonderLastkingHi there, i'm using malli 0.6.1 to do some validation in a front-end (cljs)/back-end(clj) application.
So I have a .cljc namespace where I do the validations. While this works right with clj, when I use it in cljs I see a warning message
Wrong number of args (3) passed to malli.core/-fail!
#2021-09-1916:34ikitommifixed in master#2021-09-1916:28ikitommiclosed the performance Issue (https://github.com/metosin/malli/issues/513) - schema creation & transformation is now mostly an order of magnitude faster:
(def schema
[:map
[:x boolean?]
[:y {:optional true} int?]
[:z [:map
[:x boolean?]
[:y {:optional true} int?]]]])
(def schema (m/schema ?schema))
;; 44µs -> 3.4µs (13x)
(bench (m/schema ?schema))
;; 4.2µs -> 830ns (4.5x)
(bench (mu/assoc schema :w :string))
;; 134µs -> 15µs (9x)
(bench (mu/merge schema schema))
;; 51µs -> 3.9µs (13x)
(bench (mu/closed-schema schema))#2021-09-1916:31ikitommi… and opened a new one, to add first-class support for the (compact) map-syntax: https://github.com/metosin/malli/issues/543. This is really important in for building large schema systems a) via inferring or b) to be run on slow js-runtimes. Initial design (which is already 15x faster 🚀)
(def ?schema
[:map
[:x boolean?]
[:y {:optional true} int?]
[:z [:map
[:x boolean?]
[:y {:optional true} int?]]]])
(m/form ?schema)
;[:map
; [:x boolean?]
; [:y {:optional true} int?]
; [:z [:map
; [:x boolean?]
; [:y {:optional true} int?]]]]
(m/ast ?schema)
;{:type :map,
; :keys {:x {:order 0
; :value {:type boolean?}},
; :y {:order 1, :value {:type int?}
; :properties {:optional true}},
; :z {:order 2,
; :value {:type :map,
; :keys {:x {:order 0
; :value {:type boolean?}},
; :y {:order 1
; :value {:type int?}
; :properties {:optional true}}}}}}}
(-> ?schema
(m/schema) ;; 3.7µs
(m/ast) ;; 1.1µs
(m/schema) ;; 250ns (15x)
(m/form)
(= (m/form ?schema)))
; => true#2021-09-1916:34ikitommiit’s still fully generic, like the mu/to-map-syntax & mu/from-map-syntax , but using a new support protocol, schemas can do whatever they want. The new ast can be used as “I know what I’m doing” kinda way => no parsing & checks needed.#2021-09-2016:04ikitommi(assert
(= (m/form [:=> [:cat :int] :int])
(m/form {:type :=>
:input {:type :cat
:children [{:type :int}]}
:output {:type :int}}))
"malli supports both hiccup & map syntax")#2021-09-2016:06ikitommihttps://github.com/metosin/malli/pull/544/commits/e44f8c027699a63f73b941137151e5a489501587#2021-09-2107:35eskosIs this unified in the sense that all hiccup gets internally turned into maps and then whatnot or two separate paths? Just trying to think onwards whether this could actually be a cause of sneaky conversion bugs in the long run…#2021-09-2107:36eskos(I haven’t looked into malli internals in ages so judging a single commit diff doesn’t answer that to me)#2021-09-2205:30ikitommithere is just one path, currently the map-syntax (the new ast) is transformed into properties children and thus, the old code path is used.#2021-09-2205:32ikitommiI’ll most likely reverse that, so the default path is the AST, just a hiccup->ast converter is just mechanical.#2021-09-2205:34ikitommiwhy? the ast-path is much faster (with maps), as there is no parsing needed. if real life projects, seen 10x diffrence, 100ms vs 1sec to load all the hundreds of schemas in a slow device.#2021-09-2205:35ikitommialso, the example was broken, the conversion was nor recursive, fixed inline.#2021-09-2306:12Ben SlessI know there's a lot of focus on performance at the moment but what's the status of json-schema and the missing bits, especially date/time?#2021-09-2308:36ikitommigood question.
• JSON Schema -> malli, have been waiting for the PR to progress, might be stalled
• Date/time, @henryw374 might have an insight on the new js/temporal lib status. If there would be a good set of abstractions for clj/s, malli should use them. Or, just a new alpha ns from the existing clj-time things
• others: recursive generation, proper inferring, schematized defn, typescript-compat, map+map-of, local registries refined, ... lot of things to do, some things are started by various contributors, many just ideas. After the ast-cleanup, have clojurists-together -time to work on many things, but we (all) could do design for the missing features and mark them as "PR welcome".#2021-09-2311:37Ben SlessI hope you don't mind, I went ahead and opened a draft PR to add time schemas in clj. I think it's 90% there and I'd rather have 90% to work with vs. 100% in time t > T#2021-09-2311:38Ben Slesslooking at json-schema -> malli gave me a headache, because the schema is versioned! do you want to support "everything" or just latest?#2021-09-2311:39Ben SlessThis has real implications, for example, when I was wrangling vega in #datavis malli could have been a great help but vega uses an old version of json schema#2021-09-2308:39ikitommiI tried to categorize the issues, but could add more explicit tags on those that are discussed already and would be ready to be picked up for anyone wanting to contribute.#2021-09-2308:40ikitommithe ast-change is fundamental, planning to finish a first version of that now. Few days of work I would guess, mainly tests.#2021-09-2311:41CarloI know of aave and snoop for instrumentation, is there a library in the malli ecosystem that does automatic generative testing of functions?#2021-09-2418:49Tiago Dall'OcaHow to get the original form of a schema-id? Details in the thread#2021-09-2418:49Tiago Dall'Oca#2021-09-2418:50Tiago Dall'Oca#2021-09-2418:50Tiago Dall'OcaIt's strange to me that m/form works for unqualified keys#2021-09-2418:50Tiago Dall'OcaHow do I get this to work for qualified keys too?#2021-09-2419:06Tiago Dall'Oca#2021-09-2419:10Tiago Dall'OcaOk, some more source code reading did it haha#2021-09-2419:11Tiago Dall'Ocaoptions for adding malli schemas to jsdoc#2021-09-2512:17ikitommilooks good imo. what is the status of the ts-mappings @mrdalloca? Is there something you need help with?#2021-09-2513:03Tiago Dall'OcaI'm currently focused on function typing and figuring how to make malli's functions (explain, validate, etc) available to js consumers#2021-09-2513:04Tiago Dall'OcaThen there's some refactoring to do, to make the source code more presentable#2021-09-2513:05Tiago Dall'OcaExternal types (from other libs) will soon become a priority#2021-09-2602:11Carlousing malli/core, is there a way to spec a plus function over positive numbers, asserting that that the result is >= of any of the arguments?#2021-09-2610:14Carlothe closest thing I saw is using the :fn spec to relate the arguments, but I'm not sure how to involve the result of the function#2021-09-2611:00lsenjovI’m not sure you actually can, off the top of my head. You could use pre-post maps in core clojure to test these contracts#2021-09-2611:02lsenjov(If there’s a way to do it I’m sure someone will have a peek when the working week starts)#2021-09-2611:23ikitommi@UA7E6DU04 no, there is no :fn, like there is for spec:
(s/fdef ranged-rand
:args (s/and (s/cat :start int? :end int?)
#(< (:start %) (:end %)))
:ret int?
:fn (s/and #(>= (:ret %) (-> % :args :start))
#(< (:ret %) (-> % :args :end))))
, but I don’t see any reason why it would not be supported.#2021-09-2611:24ikitommiplease write an Issue, so it’s on the backlog.#2021-09-2611:25ikitommialso. there is silly assert inside malli.core that the :input (`:args` in spec) need to be a :cat, so :and doesn’t work. Easy to fix.#2021-09-2611:32ikitommiast-options for eager references, this:
[:and
{:registry {::a ::b
::b ::c
::c [:schema pos-int?]}}
::a ::b ::c]
ast options:
1️⃣ current:
{:type :and,
:children [{:type ::m/schema, :value ::a}
{:type ::m/schema, :value ::b}
{:type ::m/schema, :value ::c}]
:registry {::a {:type ::m/schema, :value ::b}
::b {:type ::m/schema, :value ::c}
::c {:type :schema, :child {:type 'pos-int?}}}}
2️⃣ less noicy:
{:type :and,
:children [{:type ::a}
{:type ::b}
{:type ::c}]
:registry {::a {:type ::b}
::b {:type ::c}
::c {:type :schema
:child {:type 'pos-int?}}}}
3️⃣ compact:
{:type :and,
:children [::a ::b ::c]
:registry {::a ::b
::b ::c
::c {:type :schema
:child {:type 'pos-int?}}}}#2021-09-2611:36Ben Sless3 looks best#2021-09-2613:02Karol Wójcik3#2021-09-2613:59ikitommiI think 3 is also the fastest, short-circuiting to lookup instead of walking more maps before the lookup. Pushed the registry into it's own key, it's also faster to use (one map lookup less) and looks better. So, ast is map with :type, optionally :properties and :registry. Plus any keys Schema wasn't to lift there. There are few comventions with helpers:
• no child's -> nothing added
• one child -> :child
• many childs -> :children
• same, but not schemas, just values (e.g. [:> 1], [:enum "small" "large"] -> :value and :values
• entries -> :keys with maps with :value, optionally :prooerties and :order
... but, could have anything, like :=> has :input & :output#2021-09-2613:59ikitommiI think 3 is also the fastest, short-circuiting to lookup instead of walking more maps before the lookup. Pushed the registry into it's own key, it's also faster to use (one map lookup less) and looks better. So, ast is map with :type, optionally :properties and :registry. Plus any keys Schema wasn't to lift there. There are few comventions with helpers:
• no child's -> nothing added
• one child -> :child
• many childs -> :children
• same, but not schemas, just values (e.g. [:> 1], [:enum "small" "large"] -> :value and :values
• entries -> :keys with maps with :value, optionally :prooerties and :order
... but, could have anything, like :=> has :input & :output#2021-09-2713:28jkrasnayHi, I have a schema that looks like this:
(def MySchema [:or
:string
[:and
[:vector any?]
[:fn {:error/fn
(fn [{:keys [value]} _]
(str "Not found: " (pr-str value)))}
my-lookup]]])#2021-09-2713:29jkrasnayWhen I run (-> (m/explain MySchema 42) (me/humanize)) the result is ["should be a string" "invalid type" "Not found: 42"]#2021-09-2713:31jkrasnayWhat I’m hoping for is a way to organize the schema so that the error is something like “should be a string or a vector”, and if the value is a vector and my-lookup fails the error is simply “Not found: [:whatever]“. Is there a way to do this?#2021-09-2713:38jkrasnay(also it would be nice if my-lookup didn’t get called if the value wasn’t a vector. I suppose [:and] does not short-circuit)#2021-09-2713:53jkrasnayI suppose I could replace my-lookup with #(or (not (vector? %)) (my-lookup %)), then put an :error/fn on the :vector check that returns “should be a vector”.#2021-09-2714:04jkrasnayStill interested if there’s a more elegant way to do this, though.#2021-09-2714:33ikitommi@jkrasnay there is a hidden feature where you can add the error-properties into parent-nodes: in case a child errors, the too-most error is used instead, once for all errors. I was not happy with the impl and didn't need it in my client project, so decided not to release it.#2021-09-2714:36ikitommitry: https://github.com/metosin/malli/blob/master/test/malli/error_test.cljc#L499-L524#2021-09-2714:58jkrasnayOK, thanks. I’ll take some time to study these.#2021-09-2714:39ikitommi:and and :or short-circuit & collect all errors - it would be easy to inject extra data to those explanations and add options to either schema itself (via properties) or the me/humanize to allow control how to work with them.#2021-09-2717:25NicholasHello there, I've just started a very basic template with Shadow-cljs and Malli , but as I run npx shadow-cljs watch :app I get an error saying : *Wrong number of args (3) passed to malli.core/-fail!* . The project itself is just here : https://github.com/bitbot123/shadow-cljs-malli/blob/master/src/core.cljs (on the master branch) , and I've stuck [metosin/malli "0.6.1"] in shadow-cljs.edn , is this correct? Any help is much appreciated.#2021-09-2718:12ikitommioh well, released version with just the minimal fixes for that @shamansandtheprimes :
➜ ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.6.2"}}}'
Downloading: metosin/malli/0.6.2/malli-0.6.2.pom from clojars#2021-09-2718:13ikitommihttps://github.com/metosin/malli/commit/98167660c841fdf9618974a83e7e7d2fa3acd5b2#2021-09-2811:30NicholasThe project compiles no problem, thank you @ikitommi#2021-09-3008:02Yehonathan SharvitMalli mentions JSON schema as a source of inspiration (in malli's https://github.com/metosin/malli#links-and-thanks).
Here is an https://blog.klipse.tech/javascript/2021/09/30/data-validation-with-json-schema.html I wrote about JSON schema validation that mentions Malli.
The loop is closed 😎#2021-10-0113:32ikitommiGood article. I think you could implement the Full JSON Schema with malli. Maybe Malli should be rebranded as an library to create schema libraries 👻#2021-10-0310:27Yehonathan SharvitWhat would it take to implement JSON Schema with malli @U055NJ5CC?#2021-10-0114:43zaneWhat is the meaning of the error :malli.core/potentially-recursive-seqex?#2021-10-0116:09ikitommiI believe it's documented on README &/ source code.#2021-10-0116:21zaneI looked through both and it’s not readily apparent to me.#2021-10-0116:34ikitommiI'll try: it's a design choice to keep the parsing fast and simpler: You can't recur from a sequence itself, but you can recur from a element in the sequence. http://malli.io has an example how hiccup works.#2021-10-0116:35ikitommie.g. wrap the :ref inside a :schema forcing it out of the secuence to it's own schema.#2021-10-0116:45zaneThat’s extremely helpful. Thanks!#2021-10-0120:21Carsten BehringI am struggling to define a schema as defn metadata, which accepts any values as inpu (as I only one to validate output)
How could this look like ?#2021-10-0122:18respatializedassuming your function is 1-arity, you could do:
(defn my-func {:malli/schema [:=> [:cat :any] output-schema]}
... ; body goes here)#2021-10-0214:53respatializedIs there an option for m/parse that disables returning tagged values for :orn / :altn / :multi / etc?#2021-10-0215:09ikitommicurrently, no. What is your use case for this?#2021-10-0215:10ikitommiif you want to disable all, just call validate and return the original value in case of success ..#2021-10-0215:26respatializedI have a large, complex schema that implements a https://github.com/fabricate-site/fabricate/blob/c04afeb0ab40b49f1f95bde126333992b25ae644/src/site/fabricate/prototype/html.clj#L150 for Hiccup elements. I leverage :orn throughout to preserve contextual information about elements (such as whether they're flow content or phrasing content).
This is what it looks like when using a parser to parse a small element:
(site.fabricate.prototype.html/parse-element [:p "text" [:em "with emphasis"]])
=>
[:flow
[:p
{:tag :p,
:attrs nil,
:contents
[[:atomic-element [:text "text"]]
[:node
[:em
{:tag :em,
:attrs nil,
:contents [[:atomic-element [:text "with emphasis"]]]}]]]}]]
While this contextual information is sometimes useful, I also want the option of returning parsed values without it: just the {:tag t :attrs {} :contents [...]} structure so that every result from m/parse is returned in a uniform way.
Perhaps, as you suggest, I'm relying too heavily on m/parse here; there's not much stopping me from pattern matching on the head & tail of the actual elements (so long as they validate) using ordinary sequence functions.#2021-10-0215:28respatializedjust as an aside, the fact that it's possible and performant to implement this much contextual information using seqexes speaks to the expressive power of malli. 🔥#2021-10-0216:29respatializedafter rewriting the schema manually, I think I was confused about how tagged entries work. if tagging were disabled for the schema, then I wouldn't even get the map syntax from the result.#2021-10-0216:30respatializedmy intended result was a mixture of tagged and untagged parse results, so it wouldn't even have been achieved by disabling tagging for :orn.#2021-10-0312:41andre.richardsEdit: I implemented below by overriding malli.json-schema/-schema. In there I can control whether to return the ref (and add it to definitions) or return the deref of the attribute schema.
Is there a way to 'deref' [:map] children when transforming a schema to Swagger?
We use 'decomplected' map attributes, e.g.:
(def registry
(merge
(m/default-schemas)
{:customer/id :uuid
:customer/name :string
:Customer [:map
[:customer/id]
[:customer/name]}))
(transform :Customer {:registry registry})
Results in:
{:type "object",
:properties {:customer/id {:$ref "#/definitions/:customer/id"},
:customer/name {:$ref "#/definitions/:customer/name"}},
:required [:customer/id :customer/name],
:definitions {:customer/id {:type "string", :format "uuid"},
:customer/name {:type "string"}}}
Apart from the fact that this gives an error in Swagger (the extra forward slash after customer in $ref value), we would really like to get Swagger that looks like this (i.e. without a definition for each attribute):
{:type "object",
:properties {:customer/id {:type "string", :format "uuid"},
:customer/name {:type "string"}},
:required [:customer/id :customer/name]}
We re-use the attributes in multiple schemas, and really like the fact that we don't have to re-define the attribute schema every time we use it (which is the point of decomplected attributes), and this is working as expected for all the core Malli operations like validate.
We considered calling m/deref-all on the :Customer schema, but [:map] does not deref its children, so this does not help.#2021-10-0411:52Yehonathan SharvitWhat is the relationships between decode and validate?
Are we suppose to validate after a value has been successfully decoded?#2021-10-0411:57Ben SlessDecode never "fails", it's a best effort to get data to look like what should be validated.
Validation should be done after decode#2021-10-0411:58Ben Slessafter decode and before validation you can't really know if the data is valid, yet#2021-10-0412:03Yehonathan SharvitMakes sense @UK0810AQ2 . Thanks#2021-10-0416:01Carsten BehringHow can I do a schema, which allows maps of "any key" of a certain type , string for example (or qualified symbols)#2021-10-0416:08Carsten Behringfound it, :map-of#2021-10-0423:23onetomis anyone aware of any work being done on OpenAPI (v2/v3) to Malli conversion?
we are using https://github.com/oliyh/martian to talk to many APIs, and we love it, but it's using Plumatic Schema and we would like to use Malli instead.#2021-10-0423:53onetomim just watching https://www.youtube.com/watch?v=gMYQ1vDy7d0 and learnt that OpenAPI v3.1 is gonna be fully compatible with JSON Schema draft 8#2021-10-0509:54Carsten BehringI could not get my head arround the best practices to use Malli for runtime checking of function parameters.
I saw the docu about this, but wonder between "instrumentation", which seems to be dev oriented,#2021-10-0509:55Carsten Behringvs call validate in the code#2021-10-0509:55Carsten BehringI store my schema as function metadata if this helps.#2021-10-0509:56Carsten BehringHow do people do this ?#2021-10-0514:40danielglauserWhen working with Malli we used it to test all API inputs to the system and sometimes the outputs from the system. We didn't use it to check common function calls. Why do you want to do that?#2021-10-0514:50Noah Bogartfor development it’s very nice to see when you meant to pass in a vec and passed in a map instead, but in prod i feel like those kinds of checks aren’t helpful (especially if the code actually runs how you expect anyways)#2021-10-0515:15danielglauserI can see that though I usually accomplish those types of checks through my tests. It'll be interesting to see in anyone has a technique for using Malli for environment based assertions.#2021-10-0515:21ikitommi@danielglauser it’s layered: malli.instrument can be used in prod too, it doesn’t contain any var-watches, clj-kondo emitting and thanks to it’s filters, you can annotate your functions with meta like {:malli/always-validate true} and instrument those in prod.#2021-10-0515:23ikitommihave worked with a lot of codebases with Plumatic Schema where few (important) functions had the :always-validate on. e.g. when reading data from document-db or just Very Important Function (handling money) where the perf penalty is ok.#2021-10-0515:25ikitommimalli.dev is built on top of malli.instrumentwith just start! and stop! , which both are stateful. would not use in prod.#2021-10-0515:27ikitommistateful as in watching the Vars and emitting (clj-kondo) files.#2021-10-0522:29Carsten BehringI think "input checking" at runtime can be important for public functions.
At the end, I only guarantee that my function works, if the caller does his part (giving valid input only).
Failing so, resulst often in very ugly error message deep in Clojure, from which it is impossible to read what I have done wrong (as teh caller).#2021-10-0522:30Carsten Behring"Output" checking is a complete different story. That I probably don't want to have in production code.#2021-10-0522:31Carsten BehringIs somebody doing this ?
I see various way to implement the parameter checking, and was wondering which works best.
1. Instrumenting
#2021-10-0522:31Carsten Behring2. direct calls of format#2021-10-0522:32Carsten BehringTogether with this come the different options, where to keep the schema ?
as defn metadata or in some vars ?
How can we the use those easely for documentation of a function ?#2021-10-0619:32agigaoHey guys, any pointers how can I check/validate a value for bigdec?#2021-10-0619:42ribelohttps://clojuredocs.org/clojure.core/decimal_q#2021-10-0619:44agigaoOh, thanks!#2021-10-0807:19HukkaWe didn't have a test for it, and managed to (of course) put JSON schema example data that didn't match the schema, so the swagger ui happily offered to make requests that gave an error. I tried to look for it, but couldn't find any: is there a ready made helper that creates a sample data structure from the embedded :json-schema/examples, or should we just walk through the schema tree and built it ourselves?#2021-10-1316:36Carsten Behringcan I easely declare in a schema that all keys are optional ?#2021-10-1318:19HukkaEasily as data, no, but there are helpers that transform a schema recursively to open or closed#2021-10-1318:19HukkaWhoops, what am I saying, that'd completely different, of course#2021-10-1318:20HukkaThere's a separate helper for opening and closing#2021-10-1318:21HukkaAnd then others for optional and required#2021-10-1319:03Carsten Behringok, thanks#2021-10-1322:30lsenjovmalli.util has basically everything you’ll need#2021-10-1322:33lsenjovSpecifically, optional-keys#2021-10-1322:33lsenjovThere’s also malli.core/children for getting all keys out#2021-10-1411:45Ben SlessI'm trying the instrument example from the documentation, while I get the report in the REPL, kondo isn't emitting the lint warnings for it. Am I missing anything?#2021-10-1411:46borkdude@ben.sless Have you added the emitted config to your clj-kondo config?#2021-10-1411:48Ben SlessLooks like I am missing something. The emitted config is written by malli by default to .clj-kondo/configs/malli/config.edn, do I need to include it somehow?#2021-10-1411:49borkdudeyeah, in .clj-kondo/config.edn you need to write {:config-paths ["configs/malli"]}#2021-10-1411:50Ben SlessWoohoo!#2021-10-1411:51Ben SlessThank you 🙂#2021-10-1813:15andre.richardsThanks, this solved it for me as well 🙏
A question on clj-kondo/malli integration: when a function takes a map as parameter, clj-kondo is highlighting if the passed parameter is not a map, but it does not highlight if the value under a certain key is not correct type. Is this expected, and a hard limitation of clj-kondo/malli, or something that just has not been done (because it will require a lot of work, etc.). I just want to know where to set my expection. 😁#2021-10-1412:02jussiHey, I ran into the following when converting a schema to PlantUML.
(def value
[:and
{:description "Non negative numeric value."
:json-schema/example 780.56}
number?
[:>= 0]])
(def simple-registry
(merge
(m/default-schemas)
(mu/schemas)
{:value value}))
(def schema-merged
(mu/closed-schema
(m/schema
[:merge
[:map
[:name string?]
[:age value]]
[:map
[:address string?]]]
{:registry simple-registry})))
(comment
(plantuml/transform schema-merged))
Now, trying to render PlantUML results in :malli.core/invalid-schema {:schema :merge}.
Does plantuml/transform support stuff outside malli.core?
The schema-merged above expands into following schema:
[:merge [:map {:closed true} [:name string?] [:age [:and {:description "Non negative numeric value.", :json-schema/example 780.56} number? [:>= 0]]]] [:map {:closed true} [:address string?]]]#2021-10-1419:51dvingoNot sure why the plantuml transform doesn't do this but if you deref the schema, it works:
(plantuml/transform (m/deref schema-merged))
#2021-10-1421:10jussiThanks! Missed that completely 🙈#2021-10-1421:29Ivan FedorovWhat’s the replacement for a set as a predicate in malli?
Analogue of (s/def :spec1 #{:a :b})
Is it
[:or [:= :a] [:= :b]] ?#2021-10-1422:26schmee[:enum :a :b] :thumbsup:#2021-10-1422:50Ivan Fedorovright! danks @U3L6TFEJF#2021-10-1615:47chaosHi, is it possible to restrict the range of values for integer predicates produced by the generator ?
For example (malli.generator/generate [:int {:min 0 :max 6}]) reduces the range to [0,6], but (malli.generator/generate [int? {:min 0 :max 6}]) , where int? is an integer predicate, does not. My actual use case is with pos-int?. Thanks#2021-10-1615:52Ben Slesspos-int? is equivalent to [:int {:min 1}]
Type schemas are more flexible than predicate schemas#2021-10-1615:53Ben SlessYou can then add upper boundary with no complications#2021-10-1616:04chaosExcellent thanks! This is what I was looking for [:int {:min 0 :gen/max 10}] ; i.e. eq to nat-int? for validation and [0,10] for the generated range#2021-10-1616:17Ben Slessyep#2021-10-1616:17Ben SlessWhere types are concerned, I prefer the type schemas (with keywords) and not the predicate schemas#2021-10-1616:18Ben SlessYou use case is a perfect example why#2021-10-1615:50Ben SlessWhen transforming multi to json schema, how about using implications?
https://json-schema.org/understanding-json-schema/reference/conditionals.html#id7#2021-10-1620:59chaosIs there a generator option to produce distinct values for a vector? for example the following schema generated some values more than once
(->> [:vector {:min 400} string?]
malli.generator/generate
frequencies
(filter (fn [[_ v]] (> v 1))))
;; => (["" 20] ["e" 2] ["7" 2])
Thanks#2021-10-1703:45Ben SlessUse a set generator then fmap vec#2021-10-1707:42chaosGreat thanks! Although this has been a little bit of a mouthful it has done the trick:
[:vector {:min 400
:gen/schema [:set {:min 10} string?]
:gen/fmap vec}
string?]#2021-10-1707:49chaos(perhaps there should be a :gen/distinct option for collections? clojure.test.check.generators does seem to provide list-distinct and`vector-distinct` to this end)#2021-10-1721:01chaosThis unfortunately does not scale very well, because I'd need to repeat the vector predicate twice, one for :vector the other for the generator's schema, which could easily get out of sync for more complicated schemas, e.g.
[:vector {:min 400
:gen/schema [:set {:min 10} [:map [:x int?] [:y string?]]]
:gen/fmap vec
}
[:map [:x int?] [:y string?]]]#2021-10-1803:45Ben SlessAt that point you might want to define your own type and use a registry, you could call it distinct vector.
Why does it need to be a vector, btw? Why can't it be. a set?#2021-10-1817:00chaosHi, it has to be a vector because this is how values are coming out from the data source. The distinct values are only needed by the generator to produce some specific values during testing only.
I can't seem to create a custom type with -simple-schema for distinct-vector ... The syntax of the type looks to be in accordance with the section on the readme file, but it gives me a :malli.core/invalid-schema {:schema (#object[cljs$core$string_QMARK_])} error when I try to use it ...
(let [vector-distinct
(malli.core/-simple-schema
(fn [_ schema]
(let [schema (first schema)])
{:type :vector-distinct
:pred vector?
:type-properties
{:gen/gen (clojure.test.check.generators/vector-distinct (malli.generator/generator schema))}}))]
(-> [vector-distinct {} string?]
malli.generator/generate))
Thanks!#2021-10-1818:17Ben SlessI think you want a variation on vector-of#2021-10-1818:17Ben Slessthe schema itself is incorrect, not the generation#2021-10-1819:15chaossorry, not sure what you mean by "a variation on vector-of"; but I think I got it this time working, It appears I was missing the number of arguments accepted by the new type, in my case 1 (as in :min and :max):
(let [vector-distinct
(malli.core/-simple-schema
(fn [_ schema]
(let [schema (first schema)]
{:type ::vector-distinct
:min 1
:max 1
:pred vector?
:type-properties
{:gen/gen (clojure.test.check.generators/vector-distinct (malli.generator/generator schema))}})))]
(->> [vector-distinct string?]
malli.generator/generate))
Does this look sane or is it just too hacky?#2021-10-1906:19chaosActually that is not complete because pred is not going to check the vector values for conformity. I will stick with the following hack which almost does what I want (the only glitch is that generated distinct values are likely to be less than :min):
[:vector {:min 400 :max 400
:gen/fmap #(into [] (set %))} string?]#2021-10-1907:02Ben Slessah, you need a collection schema, that's it#2021-10-1814:16andre.richardsThe malli pretty printer (virhe) is failing when it tries to print a datomic db.
It might not be a defect in malli/virhe (see details below), so is there a way to exclude certain fields from pretty printing as a workaround?
I'm starting malli.dev with
(malli.dev/start! {:report (pretty/reporter)})
Take for example this function:
(defn fetch-user
"Fetches a user by its entity `id`. If no entity is found, nil is returned."
{:malli/schema
[:=> [:cat :int :any]
[:maybe :map]]}
[id db]
...)
The db (second) parameter is defined as :any - I'm not even interested in checking that it is a valid db right now, I just want to ensure the id is of valid type.
When I call the function with a valid id (:int) and db: (fetch-user 96757023244461 db), it works as expected. However when calling it with a string or anything else in the first position (`id`), it results in this:
(fetch-user "96757023244461" (db/current))
Execution error at malli.dev.virhe.EdnPrinter/visit_set (virhe.cljc:80).
Unable to convert: class datomic.core.btset.BTSet to Object[]
i.e. it is not printing the malli error as it should, but instead it fails with the above.
Passing anything else in the second argument, and still the (incorrect type) string in first argument, prints the malli error as expected:
-- Schema Error ---------------------------------------------------
Invalid function arguments:
["96757023244461" nil]
Input Schema:
[:cat :int :any]
Errors:
{:in [0],
:message "should be an integer",
:path [0],
:schema :int,
:type nil,
:value "96757023244461"}
I have attached the stacktrace for when the error can not be pretty printed (i.e. when an actual datomic db is passed).
Presumably it is traversing the db object's fields(?) and fails when it gets to the datomic.core.btset.BTSet and tries to convert it to an array.
It's not clear to me why datomic's BTSet is failing at that point, and it might not be a defect in malli, but is there a way to exclude certain fields from pretty printing, in order to work around this issue?#2021-10-1815:57ikitommioh, [malli.dev.virhe.EdnPrinter visit_set "virhe.cljc" 80] <- the datomic.core.btset.BTSet is kinda like set, but not?#2021-10-1816:01andre.richardsThat's what it looks like, yeah#2021-10-1815:58ikitommi(visit-set [this x]
(let [xs (sort-by identity (fn [a b] (arrangement.core/rank a b)) x)]
(fipp.edn/pretty-coll this "#{" xs :line "}" fipp.visit/visit)))#2021-10-1815:59ikitommi@andre.richards if you can make that sort-by wotk with datomic.core.btset.BTSet without extra deps, PR most welcome#2021-10-1816:02andre.richardsI will see what I can come up with#2021-10-1816:03andre.richardsWill maybe ask on Datomic forum what is up with that BTSet#2021-10-1817:42dominicmCalling seq on your argument might work.#2021-11-0111:52andre.richardsThis works. I will open an issue and a pull request.#2021-10-2009:31ikitommihi all. there is bunch of quality PRs being held in the queue, sorry. I’m working on with the malli internals (schema ast, lifecycle, providers, registries) and want to find a good design before merging anything big in. Will review and pull the queued PRs after that. Any small fixes welcome anytime.#2021-10-2009:32Ben Sless👍#2021-10-2009:32Ben SlessNecroing a question I asked some time ago, any idea for a transformer which removes invalid keys?#2021-10-2009:33Ben Sless(given that they're optional)#2021-10-2009:34ikitommiinvalid… you could a) validate the keys when transforming or b) explain and remove based on that.#2021-10-2009:46Ben SlessI think I can build (a), the transformer has access to the parent schema#2021-10-2009:47Ben SlessAlso reminds me, I think an interleaved transformer/validator will be good#2021-10-2009:48ikitommidoing both in a single pipeline?#2021-10-2009:49Ben Slessyes#2021-10-2009:49Ben SlessThe decoder can end up doing lots of allocations#2021-10-2009:49Ben Slessyou can short circuit on it#2021-10-2009:50ikitommibut, you can’t validate before it’s transformed, right?#2021-10-2009:50Ben Slessright#2021-10-2009:50ikitommiso, it would need to happen on :leave -> it’s all transformed already?#2021-10-2009:50Ben Slessyou'll have something like a coercer which transforms and validates in one pass#2021-10-2009:50Ben Slessthink so#2021-10-2009:51ikitommihow would it allocate less if that happens after the transformation?#2021-10-2009:51ikitommiwouldn’t all the nested childs get re-validated when you are leaving them?#2021-10-2009:52ikitommi[:map [:a [:map [:b [:map [:c [:map [:d :boolean]]]]]
#2021-10-2009:53ikitommiunless the walking knows which childs are already transformed & validated.#2021-10-2009:54Ben Slesshm, generally, you have no way of knowing if you need to re-walk the children#2021-10-2009:54Ben Slessespecially if you do interesting transformations#2021-10-2009:56ikitommimy assumption is that having validation and transformation as separate steps is the fastest way to do it. 2 simple sweeps instead of one (more complex) sweep.#2021-10-2009:57ikitommibut, all ears if there is a better way.#2021-10-2009:59ikitommioriginally, i though of doing all the workers using just -walk. there would be walker to create a validator, decoder etc. but as all schemas should have all of those and as performance was a one of the primary goals - they got separate (protocol) methods.#2021-10-2010:00ikitommithe one walker would have allowed to compose a chain of validate + transform in an easier way, I think.#2021-10-2010:04Ben SlessFigured out how to strip invalid optional keys, feel free to add it to tips, after some beautifying
(defn strip-invalid-optional-keys-transformer
([]
(let [transform
{:compile
(fn [schema _]
(let [entries (filter #(:optional (m/properties (second %))) (m/entries schema))
fs (map (fn [[k v]]
(let [validator (m/validator v)]
(fn [m]
(if-let [e' (find m k)]
(let [v' (val e')]
(if (validator v')
m
(dissoc m k)))
m))))
entries)]
(reduce comp fs)))}]
(mt/transformer
{:decoders {:map transform}
:encoders {:map transform}}))))
#2021-10-2010:04Ben Sless(m/decode [:map [:a {:optional true} int?] [:b {:optional true} int?]] {:a 1 :b 2.2} strip-invalid-optional-keys-transformer)
#2021-10-2010:05ikitommi👍#2021-10-2009:34ikitommia) might be cleaner (and faster)?#2021-10-2009:36ikitommi:or does also validation on transformation, as it needs to find the branch, which is valid after transformation.#2021-10-2009:38ikitommispike about caching computations (`-form`, -validator etc) with schemas:
(def schema
(m/schema
[:map
[:x boolean?]
[:y {:optional true} int?]
[:z [:map
[:x boolean?]
[:y {:optional true} int?]]]]))
;; 1.5µs -> 11ns (130x)
(p/bench
(m/validator schema)
;; 1.7µs -> 64ns (25x)
(p/bench
(m/validate schema {:x true, :z {:x true}})) ; => true#2021-10-2009:41ikitommithere is the initial cost of creating the thing, but just once opposed to every call. the results are cached with the actual schema instance, so when the schema instance is not needed, the cached results will also be GCd, so not leaking memory.#2021-10-2009:41ikitommiusing computation directly is still fastest, but not much:
;; 55ns
(let [validate (m/validator schema)]
(p/bench
(validate {:x true, :z {:x true}})))#2021-10-2012:27Ben SlessPlease disregard last message, user error#2021-10-2115:53Ivan FedorovHeyy, do you already have hiccup / reagent form generation on malli?#2021-10-2118:59escherizeHi Ivan, depending on what you mean, I took an experimental crack at something like that a while back. it’s really buggy but you can see it live here: https://escherize.com/w/data-desk/#2021-10-2119:17Ivan Fedorovoh niccceee!#2021-10-2119:18Ivan Fedorovopen source?#2021-10-2119:23Ivan FedorovFound it! thanks a lot!#2021-10-2209:50ikitommilooks great!#2021-10-2118:59escherize[:map [:a-single-int int?]
[:a-single-string string?]
[:a-single-bool boolean?]
[:int-vekdor [:vector int?]]
[:string-vekdor [:vector string?]]
[:bool-vekdor [:vector boolean?]]
[:map-vekdor [:vector [:map [:a-single-int int?]
[:a-single-string string?]
[:a-single-bool boolean?]
[:int-vekdor [:vector int?]]
[:string-vekdor [:vector string?]]
[:bool-vekdor [:vector boolean?]]]]]]#2021-10-2118:59escherizeif you paste that into the box on top, on blur the page should change into a ui to generate data that matches that schema#2021-10-2120:18Andrés RodríguezIs there a transformer to modify a single specific value based on its key path?#2021-10-2209:50ikitommi@hello525 sure, you could
1. put a transformer into top-level schema using schema properties as do update-in to the value
2. m/walk the schema and add the paths as schema properties to each schema. With this + :compile in transformer, you can select the schemas that need to changed at transformer creation time. There is malli.util.subschemas which could be looked as example imp how to find paths for each (nested) schema element#2021-10-2211:56martinklepschIs there a function like validate that will throw instead of returning a value?#2021-10-2212:15ikitommiat the moment, no. If that would he added, there would be a reason to add throwing validator, explain, explainer, parse, parser, transform, transformer too#2021-10-2213:34martinklepschCool, it’s fine really, just wanted to check 🙂#2021-10-2214:06Karol WójcikHowever you can create such function yourself 😄#2021-10-2214:07Karol WójcikSomething like:
(defn- explain!
[aschema x fn-name]
(when-let [info (m/explain aschema x)]
(when (or cfg/BROWSER? cfg/TEST_ENV?)
(throw (ex-info "[Malli] Schema does not match the argument"
{:type :malli-error
:x (form->pprint-str x 3)
:fn-name fn-name
:info (form->pprint-str (me/humanize info) 3)
:schema-name (schema-name aschema)
:schema-doc (schema-doc aschema)
:schema (schema->form-pprint-str (schema-form aschema))})))))#2021-10-2214:34ikitommisome perf numbers 0.6.1 vs 0.7.0 (unreleased):
(def ?schema
[:map
[:x boolean?]
[:y {:optional true} int?]
[:z [:map
[:x boolean?]
[:y {:optional true} int?]]]])
;; 44µs -> 2.9µs (15x)
(p/bench (m/schema ?schema))
;; 240ns (180x, parses entries when actually needed)
(p/bench (m/schema ?schema {::m/lazy-entries true}))
(def schema (m/schema ?schema))
;; 1.7µs -> 64ns (25x)
(p/bench (m/validate schema {:x true, :z {:x true}}))#2021-10-2509:21Ben Slesshow should I compile a transformer which I want to operate only on leave?#2021-10-2511:27Ben SlessIt's pretty terrible but it works. Anything here a bad idea?
(defn -strip-invalid-optional-keys-transformer
[schema _]
(let [entries (filter #(:optional (m/properties (second %))) (m/entries schema))
fs (map (fn [[k v]]
(let [validator (m/validator v)]
(fn [m]
(if-let [e' (find m k)]
(let [v' (val e')]
(if (validator v')
m
(dissoc m k)))
m))))
entries)]
(reduce comp fs)))
(def strip-invalid-optional-keys-transformer
(let [transform {:compile
(fn [schema _]
{:leave -strip-invalid-optional-keys-transformer})}
encoders {:map transform}]
(mt/transformer
{:decoders encoders
:encoders encoders})))#2021-10-2516:51ikitommiwith a quick look, looks good. the runtime is bare minimum, everything possible has been pushed to creation time.#2021-10-2516:51ikitommibut, should the encoding happen at enter?#2021-10-2516:52ikitommie.g. you have valid value, after encoding, the keys could be different, (e.g. stringified) so the transforming functions misses all the keys.#2021-10-2518:49Ben Slessyou're correct, I didn't consider the encode case#2021-10-2518:50Ben SlessI must say the order where things happen with regards to transformers compilation was very hard to follow 😞#2021-10-2511:27Ben SlessIt's pretty terrible but it works. Anything here a bad idea?
(defn -strip-invalid-optional-keys-transformer
[schema _]
(let [entries (filter #(:optional (m/properties (second %))) (m/entries schema))
fs (map (fn [[k v]]
(let [validator (m/validator v)]
(fn [m]
(if-let [e' (find m k)]
(let [v' (val e')]
(if (validator v')
m
(dissoc m k)))
m))))
entries)]
(reduce comp fs)))
(def strip-invalid-optional-keys-transformer
(let [transform {:compile
(fn [schema _]
{:leave -strip-invalid-optional-keys-transformer})}
encoders {:map transform}]
(mt/transformer
{:decoders encoders
:encoders encoders})))#2021-10-2515:37Ben SlessIt seems that most times where relations about map keys are requires by users it is to specify mutual exclusion. Could it be worth it to add a mutex property which takes a collection of keys which are mutually exclusive?#2021-10-2516:46ikitommiwoudn’t the key-relations solve that? havan’t had time to give that much though, but agree something should be there in the core.#2021-10-2518:49Ben SlessGetting the key-relations to work correctly and with good performance will be lots of work which I don't know when I'll get to. Adding :mutex [[:a :b]] can be done in a couple of hours#2021-10-2616:12Ben SlessFYI regarding JSON schema, might be worth to chime in https://github.com/json-schema-org/community/discussions/70#2021-10-2712:48ikitommilooks interesting#2021-10-2712:48ikitommithanks for sharing.#2021-10-2712:51Ben SlessMight be a good idea to join their slack, too. Maybe spec and malli can give them ideas#2021-10-2712:51ikitommi👍#2021-10-2712:44Ivan FedorovHas anyone seen malli to datomic schema translation projects?#2021-10-2713:31winsomeThere was a pretty good talk about using malli for this from summer 2020, I think.#2021-10-2713:31winsomehttps://www.youtube.com/watch?v=ww9yR_rbgQs#2021-10-2713:31winsomeIt's not just malli->datomic, but IIRC there's enough information here to figure it out.#2021-10-2713:35Ivan Fedorov@U028BUU1P3R talk is awesome, thanks! Is there any underlying open source?#2021-10-2713:36Ivan FedorovI have the experience and the will to write such generator, a question is if I’ll be duplicating someone’s efforts#2021-10-2713:53winsomeI don't think there is any source shared from this talk, but I was able to get something similar working myself but with datahike. I ran into some trouble getting the described "ref" system working, but even so I was able to get a pretty decent result.#2021-10-2717:47pithylessNot sure how much is covered, but there has been some work done for eql and specs: https://github.com/dvingo/malli-code-gen#2021-10-3118:06ikitommiI guess everyone is bored on the benchmarks, but updating docs, added plumatic schema to the benchmark comparison. The code looks legit :thinking_face:
(def data {:x "true", :y "1", :z "kikka"})
(def expexted {:x true, :y 1, :z "kikka"})
(spec/def ::x boolean?)
(spec/def ::y int?)
(spec/def ::z string?)
;; clojure.spec 19µs
(let [spec (spec/keys :req-un [::x ::z] :opt-un [::y])
transform #(st/coerce spec % st/string-transformer)]
(assert (= expexted (transform data)))
(cc/quick-bench (transform data)))
;; plumatic schema 2.2µs
(let [schema {:x schema/Bool
(schema/optional-key :y) schema/Int
:z schema/Str}
transform (sc/coercer schema sc/string-coercion-matcher)]
(assert (= expexted (transform data)))
(cc/quick-bench (transform data)))
;; idiomatic clojure 290ns
(let [transform (fn [{:keys [x y] :as m}]
(cond-> m
(string? x) (update :x #(Boolean/parseBoolean %))
(string? y) (update :y #(Long/parseLong %))))]
(assert (= expexted (transform data)))
(cc/quick-bench (transform data)))
;; malli 72ns
(let [schema [:map
[:x :boolean]
[:y {:optional true} int?]
[:z string?]]
transform (m/decoder schema (mt/string-transformer))]
(assert (= expexted (transform data)))
(cc/quick-bench (transform data)))#2021-10-3119:08Ben Slessnow just waiting for the protocols to move namespace and time schemas 🙂#2021-10-3118:08ikitommibtw, got the AST registry references working, could push 0.7.0 as soon as have verified it works on real projects. Then, away from perf, back to features.#2021-10-3119:30Ben Slessbtw, I have a pretty extensive getting started guide which is probably ready for initial contribution. Which format would you like it in? adoc/md/org?#2021-10-3121:36ikitommilooking forward to this! If the docs would be part of the repo, then adoc / md, so that github & cljdoc can render these. I have wanted to move to adoc, but hadn’t had time to learn the syntax differences….#2021-11-0104:14Ben SlessI'll just use pandoc to convert
It's supposed to be a superset of md#2021-10-3121:41ikitommi[metosin/malli "0.7.0-20211031.202317-3"] is out, perf + new Schema AST. Will run it against work projects next week before making a real release. Has breaking changes, mostly in the extender API, so please read the https://github.com/metosin/malli/blob/master/CHANGELOG.md#070-20211031202317-3-2021-10-31.#2021-11-0116:58Felipe Cortezhi! the doc says m/=> can be placed both before or after a defn. when putting it before, clj-kondo complains about the symbol being unresolved. :unresolved-symbol {:exclude [plus]} in clj-kondo's config.edn silences the error. should that be the default behavior?#2021-11-0116:59borkdudethat's usually not an optimal solution to unresolved symbols in clj-kondo#2021-11-0116:59borkdudedoes malli export hooks and/or config as part of the library?#2021-11-0117:00Felipe Cortezit seems to export the stuff that goes to .clj-kondo/configs/malli/config.edn automatically#2021-11-0117:03borkdudeI don't think malli exports any macro config. Does it have macros that clj-kondo doesn't understand?#2021-11-0117:05Felipe Cortezthe only thing it seems to export for m/=> is
{:linters
{:type-mismatch {:namespaces {brincando {plus {:arities {2 {:args [:int :int], :ret :int}}}}}}}}
but since the macro would still be out of order even if defined for clj-kondo, is there any other way to solve this?#2021-11-0117:06borkdudePlease give an example, I find it hard to imagine what you're talking about#2021-11-0117:07Felipe Cortezsorry! this is the working code
(m/=> plus [:=> [:cat [:int {:max 10}] :int] :int])
(defn plus [x y] (+ x y))
but I just noticed that if you "expand" that macro in clj-kondo to declare plus, it solves the undeclared variable warning, right?#2021-11-0117:08borkdudeI think you can just put the m/=> expression after the defn and then clj-kondo will understand that plus is a var#2021-11-0117:09Felipe Cortezyep, but you can also put m/=> before the defn and it is supposed to work, and I'd like to have the option to put the function schema close to the top of the defn#2021-11-0117:09borkdudeI understand#2021-11-0117:10borkdudeIn this case it's better to write a hook. Optimally malli itself would bundle this in the library and export it#2021-11-0117:10Felipe Cortezcool!#2021-11-0117:10borkdudealternatively you can use {:linters {:unresolved-symbol {:exclude [(malli.core/=>)]}}}#2021-11-0117:11borkdudethis will suppress all unresolved symbols in =>#2021-11-0117:11borkdudebut that config could also be exported by malli so it would work for everyone. cc @ikitommi#2021-11-0219:49ikitommiso, definetely. so, whoever knows better what to put into malli repo so it would work, please do.#2021-11-0219:49ikitommioh, that, I can do it#2021-11-0220:05ikitommi@UKW2FUL4D https://github.com/metosin/malli/pull/559. thing is, that if that is merged, clj-kondo will not warn on cases where you define the m/=>, but not the actual function that it points to. Not good either.#2021-11-0220:06ikitommibut, that’s stops the warnings.#2021-11-0220:56borkdude@ikitommi That's a step in the right direction, but you can make it better using an exported macro hook#2021-11-0220:56borkdudebut this is more time consuming and I understand the trade-offs#2021-11-0220:57borkdudeit would be better to export it as part of the malli library though, then people don't have to run malli first to get the better linting#2021-11-0221:06ikitommia better solution / clj-kondo macro hook would be great, anyone? Just merged the initial fix, thanks @borkdude for the code to paste in 🙂#2021-11-0313:51Felipe Corteznice! what about a hook that interprets m/=> as (fn [sym _schema] (clojure.core/declare %1))? kondo doesn't seem to complain about
(def thing)
(declare thing)
so it works the other way around too#2021-11-0317:05borkdudeI would say:
(do (declare thing) schema)
so everything you use in the schema is still seen by clj-kondo.#2021-11-0412:44Feliperight! I'll try this 🙂#2021-11-0623:40Felipe@ikitommi hmmm, apparently doesn't seem to catch errors for instrumented functions doing m/=> before the defn:
(require '[malli.core :as m])
(require '[malli.dev :as dev])
(dev/start!)
(defn plus1 [x] (inc x))
(m/=> plus1 [:=> [:cat :int] [:int {:max 6}]])
(plus1 6)
;; 1. Unhandled clojure.lang.ExceptionInfo
;; :malli.core/invalid-output {:output [:int {:max 6}], :value 7,
;; :args [6], :schema [:=> [:cat :int] [:int {:max 6}]]}
(m/=> plus1' [:=> [:cat :int] [:int {:max 6}]])
(defn plus1' [x] (inc x))
(plus1' 6)
;; => 7#2021-11-0710:28ikitommi@UA2U3KW0L that’s not good. I guess (re-)calling (dev/start!) after definitions would work.#2021-11-0710:29ikitommiso, that’s just dev-time annoyance, but then again, it’s supposed to be dev-time tooling. Ideas welcome#2021-11-0710:30ikitommiI would assume m/=> with dev running, should put a var-watcher that takes care of that, but guess not :thinking_face:#2021-11-0712:22Felipeit seems to put a watch on the function schema, so what seems to be happening is the watch fn is called, the function gets instrumented properly, but when you eval the defn again you lose the instrumentation#2021-11-0712:25Felipeyep! as a quick test, I did
(defn start!
,,,
(let [watch (fn [_ _ old new]
(println "watched")
(future ;; <-
(Thread/sleep 500) ;; <-
(mi/instrument! ,,,))]
(add-watch @#'m/-function-schemas* ::watch watch))
(mi/instrument! ,,,)
,,,))#2021-11-0713:03ikitommiso, we should get an event when the var is redefined, e.g. via defn. I guess that's doable?#2021-11-0714:23Felipe Cortezadd-watch supports this, apparently:
(def changes* (atom []))
(def a 1)
(add-watch #'a nil (fn [_ _ & old+new] (swap! changes* conj old+new)))
(def a 2)
(def a 3)
@changes* ;; => [(1 2) (2 3)]
#2021-11-0714:23Felipe Cortezjust noticed I use clojurians through 2 different slack accounts#2021-11-0117:08borkdudeor you can use {:lint-as {malli.core/=> clojure.core/def}}#2021-11-0117:08borkdudewhich is not entirely semantically correct, but I think it works for linting#2021-11-0117:09borkdudehmm#2021-11-0117:09borkdudeno#2021-11-0117:09borkdudethen you will get a redefined warning :)#2021-11-0117:09Felipe Cortezclojure.core/declare, maybe?#2021-11-0221:06ikitommia better solution / clj-kondo macro hook would be great, anyone? Just merged the initial fix, thanks @borkdude for the code to paste in 🙂#2021-11-0623:40Felipe@ikitommi hmmm, apparently doesn't seem to catch errors for instrumented functions doing m/=> before the defn:
(require '[malli.core :as m])
(require '[malli.dev :as dev])
(dev/start!)
(defn plus1 [x] (inc x))
(m/=> plus1 [:=> [:cat :int] [:int {:max 6}]])
(plus1 6)
;; 1. Unhandled clojure.lang.ExceptionInfo
;; :malli.core/invalid-output {:output [:int {:max 6}], :value 7,
;; :args [6], :schema [:=> [:cat :int] [:int {:max 6}]]}
(m/=> plus1' [:=> [:cat :int] [:int {:max 6}]])
(defn plus1' [x] (inc x))
(plus1' 6)
;; => 7#2021-11-0216:16dakraI have a beginner question:
I validate incoming JSONs by first stripping extra keys,
then validating against a malli schema and then output avro.
How can I keep/specify the type of floats/doubles?
E.g.
(-> [:map [:score double?]]
(m/encode {:score 0} mt/strip-extra-keys-transformer)
:score
type)
;; => java.lang.Long
While I want double.#2021-11-0216:21Ben SlessShould you use a string transformers first?#2021-11-0216:24dakraThis seems to work when it's a string. So
(-> [:map [:score double?]]
(m/decode {:score "0"} mt/string-transformer)
:score
type)
;; => java.lang.Double
works.. but I get {:score 0} as input and then it's still Long.#2021-11-0216:33dakraBut this lead me to look at the other transformers and decode with json-transformer seems to do the trick 🙂#2021-11-0216:37Ben SlessHang on, decode is when reading, encode is when writing#2021-11-0216:38Ben SlessIf the json is incoming you should decode#2021-11-0216:43dakraApparently I got that mixed up. But when I only use strip-extra-keys-transformer I can use encode or decode.. In both cases the extra keys got stripped. But for my (new) use-case where I want to have a double where the schema is double I have to use decode like you said.#2021-11-0217:02Ben SlessYes, where are you getting data in from? Http?#2021-11-0217:09dakraKafka#2021-11-0217:10dakraI write with jackdaw a kstreams app that validates the incoming JSON events, does some simple transformations and outputs to multiple different topics depending on the event content.#2021-11-0222:57Paul Santa ClaraHey there. I was just wondering if there were plans to expand malli to include support for openapi 3? Or should i just stick to spec-tools for the immediate future?#2021-11-0306:37HukkaThere's an issue for that where somebody was going to work on it, but I haven't got a reply last month.
That said, I heard a rumour that someone else would tackle it by Christmas. Might be that I need it sooner and might have a look too. Didn't seem terribly difficult, considering that the jsonschema support is already there. At least a simple version that is valid openapi3. Perhaps something that handles all the refs etc. would be more complex.#2021-11-0313:04Paul Santa Clarathat's great to hear that it's tentatively on the roadmap! I'll keep a close eye on things and switch over ASAP. Thanks!#2021-11-0420:31respatializedIs there a recommended way to combine parsing and decoding/transforming operations? I'm trying to parse a value and return the transformed value (there are subschemas with custom transformers) in the parsed structure.#2021-11-0506:38Ben SlessThe tldr is cps transform
Have a function which returns a parser
Every parser gets two continuations, success and failure
On providing two continuations you return the actual parsing function
Same with decode (you should validate, too while there)#2021-11-0520:50Ben SlessHere, it's a big backwards but it's the most general way of making small building blocks of it
(defn parser
[parse-fn]
(fn [success fail]
(fn [x]
(try
(success (parse-fn x))
(catch Exception e
(fail e x))))))
(defn coercer
[schema transformer]
(let [schema (m/schema schema)
decoder (m/decoder schema transformer)
validator (m/validator schema)
explainer (m/explainer schema)]
(fn [success fail]
(fn [x]
(let [x (decoder x)]
(if (validator x)
(success x)
(fail (explainer x) x)))))))
(def my-parser (parser ,,,))
(def my-coercer (coercer ,,,))
(def work
(my-parser
(my-coercer identity on-error)
on-error))
#2021-11-0507:15robert-stuttaford(def LearningPathItems
[:schema {:registry
{::items
[:map
[:items
{:optional true}
[:vector
#_(mu/union LearningPathItemBase)
[:ref ::items]]]
;; marker
]}}
::items])
#_
(m/validate
LearningPathItems
{:items [{:items [{:items []}]}]})
this works. but when i want to introduce another spec to mix into the ref'd spec (to specify lots of other keys that any :items child may have), it fails due to :malli.core/invalid-ref. am i forced to put all of those other key specifications directly in at ;; marker , or is there another way to compose it in?
I suppose i could just use Clojure to catenate two vectors but that feels dirty 😅#2021-11-0507:17robert-stuttafordthis works i guess
(def LearningPathItemBase
[:map
[...]])
(def LearningPathItems
[:schema {:registry
{::items
(vec
(concat
[:map
[:items
{:optional true}
[:vector
[:ref ::items]]]]
(rest LearningPathItemBase)))}}
::items])#2021-11-0507:18robert-stuttafordbtw the new faster mp/provider is fantastic @ikitommi!#2021-11-0710:30ikitommiDiscussion about registries here: https://github.com/metosin/malli/discussions/565#2021-11-0723:51jfntnThe README mentions malli.core/to-ast in https://github.com/metosin/malli#map-syntax but that function does not seem to be around anymore?#2021-11-0805:51ikitommiAdded a note that is is unreleased.#2021-11-1016:17rafdI'm trying to get the properties of a map entry, and struggling.
For example, given the following schema, I'd like to access the properties of :widget/id (ie. get back {:foo/bar 123}):
(def Widget
[:map
[:widget/id {:foo/bar 123} uuid?]])
I thought this would work, but it returns an invalid-schema error:
(m/properties (m.util/find Widget :widget/id))
Any suggestions?#2021-11-1818:16ikitommi@U0CLNM0N6 it returns the vector of [key ?props children] as a vector:
(def Widget
[:map
[:widget/id {:foo/bar 123} uuid?]])
(mu/find Widget :widget/id)
; => [:widget/id #:foo{:bar 123} uuid?]
(m/children Widget)
; => [[:widget/id #:foo{:bar 123} uuid?]]#2021-11-1818:16ikitommiyou can call second on it to get the props.#2021-11-1818:48rafdThanks Tommi. I did end up using second, but asked to check if there was a "malli-specific" approach.#2021-11-1818:49ikitommi👍 there are no helpers for handling entries atm. but they are tuple3 always, quite easy to work with 😉#2021-11-1107:07Yehonathan SharvitIs there a way to create a custom Malli tag that will allow me to write:
[:record
[:user-id :string]
[:num-of-purchases :int]]
And it would be the same as:
[:and
[:map-of :keyword :any]
[:map
[:user-id :string]
[:num-of-purchases :int]]
I need this because for the moment JSON decoder doesn't convert :map key strings to keywords https://github.com/metosin/malli/issues/568#2021-11-1107:15Ben SlessWhy not add your own transformer?#2021-11-1108:46juhoteperi@U0L91U7A8 you could use the json-transformer version I posted on the issue. (You might want to combine with :map-of code from the Malli.transform version)#2021-11-1108:46juhoteperiBut yeah if you create your own registry, you could define :record as :map-of + :map#2021-11-1109:56Yehonathan SharvitHow it would look like?#2021-11-1120:44Nikolas PafitisDoes generate cache the generator?#2021-11-1121:03respatializedmg/generator is what you're looking for I think#2021-11-1121:15Nikolas PafitisYes but if i skip the mg/generator, generate will create a generator, is that generator cached?#2021-11-1818:10ikitomminot yet, but in few mins it does - https://github.com/metosin/malli/pull/575/files#2021-11-1818:14ikitommimg/generator is still bit faster, as there is no cache lookup. Caching is done on Schema instance level, so calling generate with just hiccup effectively bypasses the cache.#2021-11-1818:18ikitommimerged, so generators are also cached by default (with the Schema instance):
(def schema
(m/schema
[:map
[:x boolean?]
[:y {:optional true} int?]
[:z [:map
[:x boolean?]
[:y {:optional true} int?]]]]))
(comment
;; 119µs
;; 16µs (cache generator)
(p/bench (mg/generate schema)))#2021-11-1818:18ikitommimerged, so generators are also cached by default (with the Schema instance):
(def schema
(m/schema
[:map
[:x boolean?]
[:y {:optional true} int?]
[:z [:map
[:x boolean?]
[:y {:optional true} int?]]]]))
(comment
;; 119µs
;; 16µs (cache generator)
(p/bench (mg/generate schema)))#2021-11-1322:22ArtemHi folks 👋 is there a way to write a schema for a map that may have any number of string keys, but also a fixed number of keyword keys? (It’s both [:map [:k :type]] and [:map-of :string :type] at the same time)#2021-11-1512:47ikitommicomments welcome on https://github.com/metosin/malli/issues/43#2021-11-1510:34biscuitpantsis it possible to ignore {:optional true} specs when generating values?#2021-11-1511:21biscuitpantsfor anyone wanting to do this, i found this: https://github.com/metosin/malli/blob/master/docs/tips.md#allowing-invalid-values-on-optional-keys
which i modified to do what i need!#2021-11-1511:16HukkaWhen would I want to use :sequential instead of :repeat without options? Latter would seem simpler to remember, since I have to use :repeat with min and max now and then.#2021-11-1512:45ikitommiyou can also do [:sequential {:min 1, :max 10}]#2021-11-1512:46ikitommiI would recommend using :sequential always if possible, it’s much faster and always a standalone schema. having :repeat inside of :cat makes it part of the :cat, not a standalone sequence.#2021-11-1513:18HukkaAh, I see, good to know#2021-11-1617:36EddieI am having trouble with mutually recursive schemas raising :malli.core/potentially-recursive-seqex. The intention is for the schemas to be recursive so my question is: why is the exception is being raised? I assume I have expressed recursion in an improper way, but I can’t quite figure it out. https://gist.github.com/erp12/5f7a46ff4a960feeed155da6cf89b0db the schemas and the invocation of m/validate that raises the error. Any help would be much appreciated!#2021-11-1617:44ikitommiIn App, you should wrap the :refs in :schena , like [:schema [:ref :lit]] so that they are not inlined in the :cat.#2021-11-1617:50EddieThanks, that works. I think I understand what you mean by “inlined by the :cat” but I am surprised that :ref must be wrapped. I thought only seqex schemas (`:cat` , :*, :alt, etc) had to be wrapped to prevent inlining. I am probably not understanding something about seqex in general. I’ll keep reading. Thanks again!#2021-11-1815:03rovanionI'm having trouble finding from-ast in malli.core. When I go to source in Emacs I get to 0.7.0-SNAPSHOT, so it should be the right version of the library that's loaded. But I can only see the name declared, never see it def'd.#2021-11-1815:53ikitommireleased most likely next week. You can take the latest sha from master with deps if you want to test it#2021-11-1816:02rovanionSorry, on Leiningen.
Will the AST behave like the vector based schemas? That is: Print like a vector/map but not implement ISeq/Associative?
What I'm really after is using the Malli schemas I've written to generate some Simple HTML forms.#2021-11-1816:15ikitommiSounds cool, there are multiple ways to walk the schemas for that:
1. use m/walk, look for json schema ns for example how to use it
2. use malli.util/subschemas to get the expanded list of the schema tree. Added that from a malli->form use case 😉 #2021-11-1816:15ikitommion lein, can't use AST yet, sorry#2021-11-1816:15ikitommiAST is just clojure maps#2021-11-1906:38ikitommirefreshed the latest SNAPSHOT with all stuff in:
➜ ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.7.0-SNAPSHOT"}}}'
Downloading: metosin/malli/0.7.0-SNAPSHOT/malli-0.7.0-20211118.202503-4.pom from clojars#2021-11-2209:48rovanionHmm, I get an exception thrown at me when loading malli in my project with this latest snapshot:
#error {
:cause No such var: mr/fast-registry
:via
[{:type clojure.lang.Compiler$CompilerException
:message Syntax error compiling at (tove/core.clj:1:1).
:data #:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source tove/core.clj}
:at [clojure.lang.Compiler load Compiler.java 7652]}
{:type java.lang.ExceptionInInitializerError
:message nil
:at [java.lang.Class forName0 Class.java -2]}
{:type clojure.lang.Compiler$CompilerException
:message Syntax error compiling at (malli/core.cljc:2362:53).
:data #:clojure.error{:phase :compile-syntax-check, :line 2362, :column 53, :source malli/core.cljc}
:at [clojure.lang.Compiler analyze Compiler.java 6812]}
{:type java.lang.RuntimeException
:message No such var: mr/fast-registry
:at [clojure.lang.Util runtimeException Util.java 221]}]
But I don't get the same exception when just requiering malli.core in an empty project with just malli and clojure declared as dependencies.#2021-11-1815:41Yehonathan SharvitHow can I define a custom predicate? E.g. a string of length 5 (or another more advanced logic).#2021-11-1815:41Yehonathan SharvitI tried the following#2021-11-1815:42Yehonathan Sharvit[:map {:registry {:asset-id [:and :string #(= 5 (count %))]}}
[:aa :asset-id]]#2021-11-1815:42Yehonathan SharvitBut malli says that the schema is not valid#2021-11-1815:52ikitommi@viebel wrap custom fns in :fn#2021-11-1815:54ikitommialso, [:string {:max 6}]#2021-11-1909:11Yehonathan Sharvit:fn is perfect!#2021-11-1909:20Yehonathan SharvitHow would we write a schema for a string made of two components a and b separated by a / where the schema of b depends on the value of a. The valid values of a are known in advance.
For instance:
1. When a is "ip" , b should be a valid ip
2. When a is "domain", b should be a valid domain
Here are a few examples of valid and invalid data:
• ip/127.0.0.1 is valid
• ip/111 is not valid
• domain/cnn.com is valid
• domain/aa is not valid
• kika/aaa is not valid#2021-11-1910:36Ben SlessAdd a decoder which splits the string at the separator. Then it's a multi schema for a tuple which dispatches of first#2021-11-1912:33Yehonathan SharvitYou mean a custom transformer like in https://github.com/metosin/malli/blob/master/docs/tips.md#trimming-strings ?
(require '[malli.transform :as mt])
(require '[malli.core :as m])
(require '[clojure.string :as str])
;; a decoding transformer, only mounting to :string schemas with truthy :string/trim property
(defn string-trimmer []
(mt/transformer
{:decoders
{:string
{:compile (fn [schema _]
(let [{:string/keys [trim]} (m/properties schema)]
(when trim #(cond-> % (string? %) str/trim))))}}}))
;; trim me please
(m/decode [:string {:string/trim true, :min 1}] " kikka " string-trimmer)
; => "kikka"#2021-11-1912:36Yehonathan SharvitBut then, it's the responsibility of the validate callers to explicitly mention my custom transformer. Is there a way to have the custom transformer somehow embedded in the schema?#2021-11-1912:37Ben Sless(def Composite [:tuple {:decode/string #(str/split % #"/") A B])#2021-11-1912:40Ben SlessYou can do some sort of schema constructor, too:
(def Composite (-simple-schema (fn [A B] [:tuple {:decode/string #(str/split % #"/") A B])))#2021-11-1912:40Ben SlessThen use it like (Composite [:= domain] Domain)#2021-11-1912:41Ben Slesssort of type-constructors#2021-11-1912:59Yehonathan SharvitThanks @UK0810AQ2
I'll give it a try#2021-11-1913:03Ben SlessThe way I found works well for me is starting from an ideal representation then working backwards with transformers to get there#2021-11-1909:23Yehonathan SharvitIt would be nice to be able to have a distinct error message when we explain why the data is invalid. The explanation could be either:
1. Unknown value of a
2. b is not a valid ip
3. b is not a valid domain#2021-11-1909:32rovanionI don't know how malli works, but if you can use regexes to match strings you could write something like:
((ip)/(:ip-address:)|(domain)/(:domain-name:))
#2021-11-1909:33rovanionAnd then look at capture group 1 to see what a is and capture group 2 to find the value.#2021-11-1909:34rovanionIf regexes aren't available the logic that would be transferrable would be to group a and b together in pairs rather than to have separate specs for a and b.#2021-11-2213:25Yehonathan SharvitInside a map, should metadata like title and description be on the field key or in the field value?
For instance:
[:map [:id {:title "The ID"} :string]]
or
[:map [:id [:string {:title "The ID"}]]
?#2021-11-2217:35ikitommiboth work, up to you. If the values are shared, pushing the properties into value seem like a better idea, e.g.
(def id (m/schema [:string {:title "The ID"}]))
(m/schema [:map [:id id]])
#2021-11-2303:28Yehonathan SharvitIn most cases the meaning depends on the context. I think it should be given by the key .#2021-11-2217:56ikitommiabout the a/b thing… agree with Ben and that’s how we have used it. the actual domain model could be defined as :multi and there would be encode & decode to&from the string-domain. Actual domain-side would be something like: https://github.com/metosin/malli/blob/master/docs/tips.md#dependent-string-schemas#2021-11-2218:09ikitommiping @viebel.#2021-11-2308:56Yehonathan SharvitThank you @U055NJ5CC for adding a section about string dependent types in malli documentation.#2021-11-2222:05respatializedI'm wondering if there's something I'm not quite understanding about how m/decode and transformers work in the context of :or:
(m/decode
[:cat {:decode/get {:leave second}} [:= :txt] :string]
[:txt "something"]
(mt/transformer {:name :get}))
; => "something"
(m/decode
[:or [:cat {:decode/get {:leave second}} [:= :txt] :string]
:double]
[:txt "something"]
(mt/transformer {:name :get}))
; => [:txt "something"]
what I'd like is for the :or to decode its value using the decoder provided by the matching schema. I had assumed that decoding was recursive by default, but it doesn't seem to be working in the context of the subschema here.#2021-11-2305:52Ben SlessI think if you really wanted correct results here you had to write some sort of backtracking search algorithm and end up implementing half of minikanren#2021-11-2315:52respatializedinterestingly, encode does the right thing here:
(m/encode
[:or [:cat {:encode/get {:leave second}} [:= :txt] :string]
:double]
[:txt "something"]
(mt/transformer {:name :get}))
;; => "something"
what I may have been confused about was just the respective purposes of encoders and decoders#2021-11-2321:44ikitommiIt works as expected, despite it looks odd.
• Decode is a process of turning (potentially invalid) values from external format (e.g. JSON) into values that are valid in the default/clojure domain.
• Encode is the opposite: turning valid values from default domain into (potentially invalid) values in another domain (e.g. JSON).
here, your decoder is not working correctly, as it emits invalid values. The :or picks the first branch that produces valid values after the transformation. In the example, both paths produce invalid value, so the original value is returned. See the impl: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L713-L724#2021-11-2321:44ikitommiIt works as expected, despite it looks odd.
• Decode is a process of turning (potentially invalid) values from external format (e.g. JSON) into values that are valid in the default/clojure domain.
• Encode is the opposite: turning valid values from default domain into (potentially invalid) values in another domain (e.g. JSON).
here, your decoder is not working correctly, as it emits invalid values. The :or picks the first branch that produces valid values after the transformation. In the example, both paths produce invalid value, so the original value is returned. See the impl: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L713-L724#2021-11-2308:59Yehonathan SharvitMalli decoders are very cool. But as far as I understand, in order to use a schema that relies on decoders, clients have to decode data before validating it. It means that a schema that relies on decoders cannot be used as-is for validating data.
Am I correct?#2021-11-2309:19ikitommiFor now, that’s true. There was a discussion some time ago to add :parsed schema element. That would just run m/-parse in before validate, explain, transform etc. like s/conform on spec. Could also be :decoded, so that this would work:
(m/validate [:decoded {:decode :string} schema] "ip/127.0.0.1") ; => true#2021-11-2309:22ikitommiyou can do that “easily” in the user space too. Just a wrapper-schema basically.#2021-11-2312:36Yehonathan Sharvit@ikitommi Here is my attempt to write a parser in user space:
(defn my-parse
[schema data]
(let [data' (decode schema data (transformer string-transformer default-value-transformer))]
(if (validate schema data')
(encode schema data' (transformer string-transformer default-value-transformer))
(humanize (explain schema data')))))
It does the job with asset ids but the problem is that it transform integer to strings.
(def schema
[:map
[:count :int]
[:asset-id [:multi {:dispatch first
:decode/string #(str/split % #"/")
:encode/string #(str/join "/" %)}
["domain" [:tuple [:= "domain"] domain]]
["ip" [:tuple {:error/message "Invalid IP"} [:= "ip"] ipv4]]]]])
(my-parse schema {:count 12
:asset-id "ip/127.0.0.1"})
;; {:count "12", :asset-id "ip/127.0.0.1"}
;; Oops "12" instead of 12#2021-11-2312:44ikitommijust return the original instead of encode the whole value into string-domain?#2021-11-2312:44ikitommibecause:
(m/encode :int 1 mt/string-transformer) ; => "1"
#2021-11-2312:52ikitommi… or create a custom name for the parsing transformer, e.g. :parse - it’s it a name you have defined, it doesn’t transform anything else.#2021-11-2312:52ikitommi(let [schema [:map
[:x :int]
[:y [:int {:decode/parse (partial + 10)
:encode/parse (partial * 2)}]]]
t (mt/transformer {:name :parse})]
(as-> (m/decode schema {:x 0, :y 0} t) $
(m/encode schema $ t)))
;; => {:x 0, :y 20}#2021-11-2313:13Yehonathan SharvitI cannot return the original as I would like the default values to be added by the parser.
For instance, with a schema that assigns 42 as a default value to :count
(def schema
[:map
[:count [:int {:default 42}]]
[:asset-id [:multi {:dispatch first
:decode/string #(str/split % #"/")
:encode/string #(str/join "/" %)}
["domain" [:tuple [:= "domain"] domain]]
["ip" [:tuple {:error/message "Invalid IP"} [:= "ip"] ipv4]]]]])
(my-parse schema {:asset-id "ip/127.0.0.1"})
;; {:asset-id "ip/127.0.0.1", :count "42"}
#2021-11-2313:20Yehonathan SharvitHowever, I really like the idea of a custom name#2021-11-2313:21Yehonathan SharvitWe need that as our plan is to have a company wide schema repository to be consumed by all apps#2021-11-2321:45Nikolas PafitisIs there an option for MapSchemas to skip generating specific field/s?#2021-11-2321:46ikitomminot atm, but could be. Ideas welcome#2021-11-2321:48ikitommihere’s two:
;; a) a top-level property?
[:map {:gen/fields [:x :y]}
[:x :int]
[:y :int]
[:z :int]]
;; b) entry-level property?
[:map
[:x :int]
[:y :int]
[:z {:gen/gen nil} :int]]
#2021-11-2321:50ikitommib would be 1 line change to the current impl.#2021-11-2322:04Nikolas Pafitisi guess mu/select-keys could be used#2021-11-2322:25Nikolas PafitisI might have encountered a bug:
{:example/user [:map {:fully/entity? true
:fully.entity/id :user/id}
[:user/id :uuid]
[:user/username :string]
[:user/password :string]
[:user/email :string]]}
I have this map and i merge it with default-schemas and schemas to make my registry. Validate and generate work when I use them against
(m/schema type {:registry registry})
but the following returns just nil
(m/properties (m/schema type {:registry registry}))
#2021-11-2322:40Nikolas PafitisIt seems to work if i do deref before properties#2021-11-2322:45Nikolas PafitisIs this expected behaviour?#2021-11-2420:26ikitommiIt's the current behavior and surprises me every time. References are also schemas, and usually without properties. Think like:
(def foo (with-meta {} {:a 1}))
(meta #'foo) ;=> .. no :a here#2021-11-2420:27ikitomminot 100% happy, but not sure would anything more right than the current behavior#2021-11-2421:31Nikolas PafitisI guess when passed to malli functions like m/properties the different schema types should be dereffed if required. Like in nil punning where nil "can take the correct shape" depending on context#2021-11-2518:32ikitommiwe would assume the references don't have properties, but they could. We could enforce them not to have, but not sure if that would be a good design decision :thinking_face:#2021-11-2420:26ikitommiIt's the current behavior and surprises me every time. References are also schemas, and usually without properties. Think like:
(def foo (with-meta {} {:a 1}))
(meta #'foo) ;=> .. no :a here#2021-11-2518:32ikitommiwe would assume the references don't have properties, but they could. We could enforce them not to have, but not sure if that would be a good design decision :thinking_face:#2021-11-2416:15respatializedanother encode question: how do I handle subsequences that I want to encode into single values? I've got some data like:
(let [data ["some text" "ex1.csv - file 1" "ex2.csv - file 2"]
expected-result ["some text" {"file 1" "ex1.csv" "file 2" "ex2.csv"}]]
(m/encode
[:catn [:txt [:= "some text"]]
[:subseq [:repeat {:encode/extract {:leave #(apply merge %)}}
[:re {:encode/extract {:enter #(let [[id fnum] (clojure.string/split % #"\s-\s")]
{fnum id})}}
#".* - file \d"]]]]
data
(mt/transformer {:name :extract}))
)
;; => ["some text" {"file 1" "ex1.csv"} {"file 2" "ex2.csv"}]
the #(apply merge %) encoder works to merge the maps extracted if the :repeat schema is a top-level schema, but not if it is a schema for a subsequence of a seqex.
am I pushing encode too far here? should I be relying on parse to do this kind of grouping?#2021-11-2421:17respatializedthinking about this a bit more, it doesn't seem like this is possible, as different values are being encoded depending on whether the sequex is top-level or a subsequence:
top-level:
(let [data ["ex1.csv - file 1" "ex2.csv - file 2"]
expected-result {"file 1" "ex1.csv" "file 2" "ex2.csv"}]
(m/encode
[:repeat {:encode/extract {:leave #(apply merge %)}}
[:re {:encode/extract {:enter #(let [[id fnum] (clojure.string/split % #"\s-\s")]
{fnum id})}}
#".* - file \d"]]
data
(mt/transformer {:name :extract}))
)
;; => {"file 1" "ex1.csv", "file 2" "ex2.csv"}
so really the object of encoding is a complete vector, the vector of maps created after each string element gets encoded.
whereas with the subsequence, it's not a clearly delineated individual value:
["some text"
#_"start of subsequence in :repeat"
{"file 1" "ex1.csv"} {"file 2" "ex2.csv"}
... #_"arbitrary number of additional repeats"
#_"end of subsequence in :repeat" ]
and thus cannot be passed in as an argument to the function referenced in :encode/extract , even if that function accepts varargs or an input sequence. so I think I need to rely on m/parse to do the grouping I am hoping for here.#2021-11-2418:26Adam HelinsIs there some way for having context sensitive transformers?
e.g. when transforming a child in a coll, being able to take into account what happened to already transformed children#2021-11-2509:11Adam HelinsIn other words, is transforming strictly context-free?#2021-11-3011:52ikitommicould you provide an minimalistic example when this would be useful. There was a discussion about this long ago, but a) there was no real-life examples for “why?” and b) context-free made performance optimization much easier. Happy to revisit, but need the a) 🙂#2021-11-3011:54ikitommithere was also a discussion just to implement all applications (validation, transforming) using walk. That was my original idea for Spec too (https://clojure.atlassian.net/browse/CLJ-2251).#2021-11-3011:55ikitommi4 years! time flies 🙂#2021-11-3011:55ikitommi> Created 11 October 2017, 22:22#2021-11-3011:52ikitommicould you provide an minimalistic example when this would be useful. There was a discussion about this long ago, but a) there was no real-life examples for “why?” and b) context-free made performance optimization much easier. Happy to revisit, but need the a) 🙂#2021-11-2504:48JohanWould malli be a good place to check if value is within a list of ~20k items ?
Build a giant [:enum] or custom fn or just not use malli for that ?#2021-11-2509:44dangercoderI would put that list of 20k items into a set instead and use clojure.core contains? function.
If you want to check that through malli you can always create a custom :fn#2021-11-2512:52JohanFor example I want to check a postcode, would you just use a malli schema to check if the value is a number and then check if it's part of the country list of postcode ?#2021-11-2515:03vinurs(def get-categories-req
[:map {:title "get-categories-req"}
[:categories [:vector {:description "category id"
:default [0 1]
} int? ]]])
hello, i define the malli schema like this, and call
it shows the error
{
"schema": "[:map {:title \"get-categories-req\", :closed true} [:categories [:vector {:description \"category id\", :default [0 1]} int?]]]",
"errors": [
{
"value": "0,1",
"in": [
"categories"
],
"message": "invalid type",
"path": [
"categories"
],
"schema": "[:vector {:description \"category id\", :default [0 1]} int?]",
"type": "malli.core/invalid-type"
}
],
"value": {
"categories": "0,1"
},
"type": "reitit.coercion/request-coercion",
"coercion": "malli",
"in": [
"request",
"query-params"
],
"humanized": {
"categories": [
"invalid type"
]
}
}
how can i make malli parse this?#2021-11-2516:57Ivan FedorovCan malli’s [:merge] work inside a registry?
this fails for me
(def registry:shopify
{:e.shopify/address1
[:map [:first_name string?]]
:e.shopify/address2
[:map [:city string?]]
:e.shopify/address-full
[:merge
:e.shopify/address1
:e.shopify/address2]})
(def schema:address-full
(m/schema
[:schema {:registry registry:shopify}
:e.shopify/address-full]))
#2021-11-2518:35ikitommiDo you have the :merge registered? It's not in the default registry (just yet at least)#2021-11-2611:47Ivan Fedorov@U055NJ5CC How would I do this? This doesn’t work at least 🙂
(def registry:shopify--
{:merge mu/merge
:e.shopify/address1
[:map [:first_name string?]]
:e.shopify/address2
[:map [:city string?]]
:e.shopify/address-full
[:merge
:e.shopify/address1
:e.shopify/address2]})#2021-11-2708:45ikitommimaps do not have order, any of the entries using merge might get resolved before the :merge. I’ll change how the default registry works, so this would be easier. Before that:
1. use mu/merge instead of :merge
2. swap the default registry with one that has the :merge
3. use composite registry, e.g. (mr/composite-registry {:merge mu/merge} {:e.shopify/address1 …}), in case the :merge should be registered before the actual schemas#2021-11-2616:01emccueWhats the best way to add documentation to a schema#2021-11-2616:01emccue[:map {:doc "A position"}
[:x {:doc "The x coordinate"} :int]
[:y {:doc "The y coordinate"} :int]]#2021-11-2616:02emccuei’m fine making something up, but if there is a “standard” schema property i would want to use it#2021-11-2619:55ikitommi:title and :description are used for JSON SCHEMA. Kinda long, but current.. standard#2021-11-2619:55ikitommihttps://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc#L16#2021-11-2713:13ikitommiallowing schema registry to be swapped without compiler/jvm-options: https://github.com/metosin/malli/pull/583. There would be a "strict" mode too, if this is too loose for someone. But might be better for the 90% of cases?
(require '[malli.core :as m]
'[malli.util :as mu]
'[malli.registry :as mr]
'[malli.generator :as mg])
;; look ma, just works
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
(mu/schemas)))
(mg/generate
[:merge
[:map [:x :int]]
[:map [:y :int]]])
; => {:x 0, :y 92}
comments welcome.#2021-11-2713:14ikitommiping @U0A5V8ZR6#2021-11-2713:16ikitommiyour sample could be:
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
(mu/schemas)
{:e.shopify/address1 [:map [:first_name string?]]
:e.shopify/address2 [:map [:city string?]]
:e.shopify/address-full [:merge
:e.shopify/address1
:e.shopify/address2]}))
(mg/generate :e.shopify/address-full)
;{:first_name "0GiD"
; :city "DMS5wc6mXcN4JCp9lJ"}#2021-11-2713:17ikitommialso, you could inject a mutable-registry as last in the registry-chain and add the schemas using custom register! fn. Mutate like a boss 🙂#2021-11-2714:07Ivan FedorovOh this looks sweet! Thank you, mr Reiman!#2021-11-2717:08pithylessLooking forward to this; I’ve ended up fighting the tooling too often trying to get the compiler options working correctly in a consistent way.#2021-11-2718:11ikitommimerged in master. a blog post and shipping 0.7.0 out.#2021-11-2714:26Karol WójcikWill malli support records?#2021-11-2714:45ikitommiwhat would that look like?#2021-11-2722:53emccue[:and [:fn #(instance? RecordType %)]
[:map [:x :int] [:y :int]]]
Is a basic way if you just need validation and not generation#2021-11-2722:55emccuei’m not clever enough to compose the :map generator with map->RecordType but i assume there is a way#2021-11-2808:46Karol WójcikI was thinking about something like schema is doing.#2021-11-2809:57ikitommim/defrecord I presume. I think Malli should have that and m/defproticol too.#2021-11-2814:25Karol WójcikCool. Looking forward to this.#2021-12-0115:53wcalderipeCan I get the schema of a :multi by passing the dispatched value to a function?
Example:
(def Multi
[:multi {:dispatch :type}
[:foo [:map [:value :int]]]
[:bar [:map [:value :string]]]])
(get-schema Multi {:type :foo}) ;; => [:foo [:map [:value :int]]]#2021-12-0117:02ikitommi@wcalderipe, maybe:
(def Multi
[:multi {:dispatch :type}
[:foo [:map [:value :int]]]
[:bar [:map [:value :string]]]])
(defn get-schema [schema value]
(some-> schema
m/properties
:dispatch
(apply (list value))
(->> (mu/get schema))))
(get-schema Multi {:type :foo})
; => [:map [:value :int]]#2021-12-0117:04ikitommiif you want the whole entry, use mu/find:
(defn get-schema [schema value]
(some-> schema
m/properties
:dispatch
(apply (list value))
(->> (mu/find schema))))
(get-schema Multi {:type :foo})
; => [:foo nil [:map [:value :int]]]#2021-12-0117:05ikitommiit’s always tuple3 of [key properties value]#2021-12-0207:38wcalderipe@U055NJ5CC thanks a bunch for the help 🙌#2021-12-0316:49Ben SlessIdea - charset predicate for string schemas (alpha, alphanumeric, etc)#2021-12-0509:18Ben Slesshttps://github.com/metosin/malli/pull/587#2021-12-0520:25ikitommiLook good. The cljs-support would be really good as I think these will be used a lot.#2021-12-0520:26ikitommiany change of adding those too?#2021-12-0520:40Ben SlessTheoretically, yes, practically, my knowledge in cljs is limited and we've seen how some of the techniques I'm used to from clj don't work well for cljs.
Also worried about code size
This will require more careful review on your end, but I don't mind giving it a shot#2021-12-0521:34Ben SlessOkay, my cljs tests are failing and setting up a REPL environment is a pain. Will leave it to tomorrow#2021-12-0609:11Ben Sless@U055NJ5CC added cljs implementation#2021-12-0610:03ikitommipartywombat commented.#2021-12-0610:15Ben SlessAgreed with most comments but I think there are some edge cases regarding blankness and nil#2021-12-0509:18Ben Slesshttps://github.com/metosin/malli/pull/587#2021-12-0716:15ikitommi,🥳#2021-12-0716:55dharriganAmazing! Such a great library to use!#2021-12-0717:00Ben SlessIt's also a pleasure to hack on#2021-12-0717:00Ben SlessIt's also a pleasure to hack on#2021-12-0721:16dev-hartmannHey folks I'm new to malli and really blown away by what it can do. I just have a small question I couldn't figure out from the readme ( maybe I'm just blind) can I parse a json Schema to a malli schema?#2021-12-0721:31Ben SlessNot yet 🙂#2021-12-0721:32Ben SlessI started working in it but it's missing a few bits and bobs on the JSON schema specification to be happy with#2021-12-0814:03Yevgeni TsodikovHey, I see that https://clojars.org/metosin/malli/versions/0.7.0T exists in Clojars.
What’s the difference between that and https://clojars.org/metosin/malli/versions/0.7.0?#2021-12-0814:21ikitommiIt's a typo release.#2021-12-0815:06Yevgeni TsodikovIt’s messing up with https://github.com/renovatebot/renovate, it thinks it’s a later release than 0.7.0 😞#2021-12-0816:17ikitommigood thing is that the contents are identical to 0.7.0.#2021-12-0816:17ikitommigood reason to ship the next version soon.#2021-12-0816:26Ben SlessEven just a point release to park on the latest artifact id#2021-12-0907:18Ben SlessThis is very cool https://github.com/erp12/schema-inference#2021-12-0911:51Ben Sless@U7XR2PZFW#2021-12-0914:43EddieThanks! I’m surprised you found this so quickly without me doing any advertising. :)
I am currently using this prototype in a research project to give a more concrete example of how something like this could be valuable. After I make that work public, I plan to make a wider pitch to the community. That said, if people have any feedback and ideas now, I would love to hear it!#2021-12-0914:44Ben SlessFirst of all, you get cool-points 😎#2021-12-0914:45Ben SlessSecond, can you share some of your design decisions? Why did you choose a bespoke representation rather than use tools.analyzer output?#2021-12-0914:46Ben SlessAnother thing I thought of while reading the code was some impedance mismatch between the type information and the expression encoding which could be unified#2021-12-0914:49Ben SlessYou could create "big lambda"
;; Type Abstraction
'[:FN [:cat T] U]
Then the abstraction and environment representations could be shared#2021-12-0914:50Ben SlessAlthough => is pretty much that, already
In its context, why do you need to tag s-vars at all? You know it's a function of type to type#2021-12-0915:02Ben SlessYou can blame someone I follow for starring your project 🙂#2021-12-0916:16EddieThis is all great. I appreciate you taking the time to take a closer look.
> Why did you choose a bespoke representation rather than use tools.analyzer output?
This is the first I’m hearing of tools.analyzer :) It seems ideal for this kind of thing. I stuck to a representation that was as close to lambda calculus and Hindley–Milner as possible to make the papers easier to translate into code. The thought of expanding the bespoke representation to cover all of Clojure gave me nightmares. Tools.analyzer might save the day.
> Another thing I thought of while reading the code was some impedance mismatch between the type information and the expression encoding which could be unified …
> Although => is pretty much that, already
Yea, figured it would be nice if everyone’s function schemas “just worked” as function types during inference without any changes, so I used :=>. I’m sure you noticed that only positional args are supported right now (via :cat). At some point I would like to try to add support for varargs. Also maybe destructuring, but that might be challenging.
Still, you raise a good point about the unnecessary separation between expressions trees and type/schema trees. As you mentioned, the tagging of :var and :s-var could be eliminated. Also the logically equivalent functions like substitute-vars and substitute-types could be made into 1 generic function. I went back and forth on this design decision a couple times. In the end, my decision to separate into 2 kinds of trees was motivated by wanting more explicit information about what went wrong when debugging these algorithms. I was brand new to type theory when I started. With hindsight, I wouldn’t choose the same design.
Based off of 1 mornings worth of googling, I think a change to tools.analyzer and a unified encoding is a good direction!#2021-12-0916:24Ben SlessThe only problem with tools analyzer is it has a lot of output which can get unwieldy, there are about 15 node types to parse there instead of 4#2021-12-0916:25Ben SlessThen you might end up needing to use another method to walk the tree, either the analyzer's walk mechanism with multi methods or something like meander#2021-12-0916:26Ben SlessIf you lift the type representation to ast nodes as well you could then write your own emitter which translates it back down to a schema#2021-12-0907:33Nechemya UngarHi all, I am new to clojure (using clojurescript) and I come form a static-typing background (typescript), I looked into bringing some order into the code base I got into, and malli was recommended to me over spec.
I wanted to say that for newcomers like me it would have been helpful if the readme actually started with introduction of the malli provider - it really helps to create a quick scaffolding for data structures.
thanks2#2021-12-0912:55amithgeorgeIn the readme, it mentions decimal? as built-in schema. Checking the relevant https://github.com/metosin/malli/blob/0.7.0/src/malli/core.cljc#L2234, decimal? is not registered as a schema. decimal? was removed in this https://github.com/metosin/malli/commit/5deae59357b2eb3fb9890282e679bb2d7fa557e3. Is it possible to add it back?
At the moment I am using decimal? in a :fn schema and that works.#2021-12-0915:31ikitommiPR welcome to add it back#2021-12-0916:24amithgeorgeI maybe missing something here. I added decimal? to predicate-schemas. My expectation was after that evaling the file with this change, the following code should work
(validate :decimal 10.9M)
But that errors out with this message
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
; :malli.core/invalid-schema {:schema :decimal}
For context,
(validate decimal? 10.9M)
works fine with the change.
What else needs to change to support :decimal . Also, do you think this is needed?#2021-12-0916:27amithgeorgeFound it. If :decimal support is to be added, then I need to add :decimal to type-schemas . I will exclude this change for now and raise a PR for only the inclusion of decimal? .#2021-12-0916:51amithgeorgePR - https://github.com/metosin/malli/pull/591#2021-12-0917:14ikitommiMerged, thanks!#2021-12-1012:07Ben SlessThinking a bit about a workable composition of transformer and validator, will it be be correct to model it via a cps transform where validation is called after leave stage for each element? It is true that a schema could transform its children arbitrarily, so how could this be done in a single pass? Feels like trying to solve the halting problem#2021-12-1013:14ikitommiwoudn't the validation be called a lot more that actually needed? e.g. leaf validates itself, one level up needs to re-validate the leaf as part of it's own validation, etc.#2021-12-1013:18ikitommithe prismatic approach: https://plumatic.github.io//schema-0-2-0-back-with-clojurescript-data-coercion#2021-12-1013:31Ben SlessIdeally, no, you assume the leaves are already correct if you made it to the parent then you just satisfy the parent condition or combinator#2021-12-1013:35Ben SlessLet's say that each coercer has a success or failure continuation. On failure, we just return, on success, we call the transformation and validation of the next value#2021-12-1013:36Ben SlessThen you need to define the combination semantics for map, sequence, vector, tuple, and, or#2021-12-1013:37Ben SlessMaybe the validation itself can be split to enter/leave#2021-12-1013:37Ben SlessMap could check on enter that it's a map, then coerce its children#2021-12-1013:40Ben SlessThis holds as long as a transformer doesnt modify children recursively#2021-12-1013:40Ben SlessWhich is true for 99% of cases? So maybe there could be an optimistic coercer which isnt guaranteed to work all the time#2021-12-1100:00steveb8nQ: I have a single arg fn that I am trying to generatively test. The single arg is vector that is checked using a :catn schema. when I use the => syntax, it interprets the nested :catn schema for the fn arity instead of treating it as a single arg. has anyone figured out how to do this?#2021-12-1100:43emccue[:cat [:cat :int :int]]#2021-12-1100:43emccueall fn args are cat#2021-12-1100:43emccueoh wait no i just realized your issue#2021-12-1103:27steveb8nYeah. Simple but not easy. I can work around it by wrapping with another fn with a map arg.#2021-12-1103:27steveb8nI'll log an issue to see if we are missing something#2021-12-1103:53ikitommiwrap in :schema to push out of the sequence#2021-12-1105:01steveb8nthx @U055NJ5CC I’ll give that a try#2021-12-1121:14steveb8nconfirming, using :schema around the vector schema fixes the problem#2021-12-1107:20Ben SlessReading @afoltzm FSM post again, I returned to the idea of some sort of schema router / compiler, where an or of schemas will pull out the common denominator then dispatch to the difference
Thoughts?#2021-12-1109:39steveb8ncan you explain more? I’ve been using them more in SPA’s so interested in this#2021-12-1110:25Ben SlessImagine [:or [:map [:a any?] [:b any?]] [:map [:a any?] [:c any?]]] => [:and [:map [:a any?]] [:or [:map [:b any?]] [:map [:c any?]]]] to start with, even smarter to have a dispatch to the correct schema instead of or#2021-12-1115:23respatializedSounds like a malli.diff namespace might be an interesting idea, because this concept may be applicable beyond just :or schemas – for example, you'd be able to compare two seqex schemas and say things like "sequence B can contain all the same elements as sequence A except for elements X and Y"#2021-12-1115:26Ben SlessI'm just getting started, [:or :int :number] => :number#2021-12-1115:27Ben Slessschemas as segments and sets#2021-12-1120:02mynomotoCan I have a link for this post? It looks interesting.#2021-12-1120:04Ben SlessSure, https://fabricate-site.github.io/fabricate/finite-schema-machines.htmlhtml
@mynomoto#2021-12-1120:45mynomotoThis is really interesting, both the post and the routing idea 👏#2021-12-1121:22ikitommijust would have needed an intersection of two map schemas yesterday.#2021-12-1121:24ikitommipluggable optimizer is a great idea, here's one stab at it: https://github.com/miikka/boolean-simplifier#2021-12-1209:32Ben Sless@U7XR2PZFW's work on inference could intersect with this as well.
Every type of/or schema can represent a set of predicates. A predicate can be a singular predicate, a segment (upper and lower bounds) or finite domain (min max)#2021-12-1219:49EddieSounds like roughly equivalent to sub-typing for structural types. If we assume collections are covariant with respect to elements, functions are contravariant with respect to arguments, and a few other assumptions, I suspect small-ish rule system could be used to implement a sub-type predicate.#2021-12-1219:52Ben SlessWe can even unpack map schemas to a closed set of predicates, being map? and a set of all the entry predicates. Same for tuples. Then conjunction and disjunction are pretty well defined and we remain in predicate land and not structure.
Not sure if it's good#2021-12-1220:10Ben SlessRoughly:
(defprotocol SchemaDomain
(-conj [this that]) ;; and
(-disj [this that]) ;; or
(-isa? [this that])
(-walk [this]))
(defrecord EntrySchema [k v]
(-conj [this that]
(EntrySchema. k (-conj v (:v that))))
(-disj [this that]
(EntrySchema. k (-disj v (:v that)))))
(defrecord PredicateSchema [p]
(-conj [this that]
(cond
(-isa? this that) this
(-isa? that this) that
:else (-and this that)))
(-disj [this that]
(cond
(-isa? this that) that
(-isa? that this) this
:else (-or this that))))
#2021-12-1119:43respatializedI've encountered an issue where sequence schemas appear not to preserve metadata of input collections:
(meta (m/decode
[:schema {:decode/get {:enter identity}} [:cat [:= :start] :map]]
(with-meta [:start {:a 2}] {:meta true})
(mt/transformer {:name :get})))
;; => nil
(meta (m/encode
[:schema {:encode/get {:enter identity}} [:cat [:= :start] :map]]
(with-meta [:start {:a 2}] {:meta true})
(mt/transformer {:name :get})))
;; => nil
is this expected behavior, or should malli preserve this information?#2021-12-1119:44respatialized(happy to file an issue on GitHub if it would be helpful)#2021-12-1120:04Ben SlessSure, https://fabricate-site.github.io/fabricate/finite-schema-machines.htmlhtml
@mynomoto#2021-12-1208:27ikitommirelated to is-a relations, malli should have more concrete type definitions in the core, e.g. pos-int? is not a type, it’s a :int with constraint of being positive. Less types to map in transformers and just better. Related:
• https://github.com/metosin/malli/issues/264
• https://github.com/clj-kondo/clj-kondo/blob/d9fca2705863e3e604e004ccb942e0b3d2e268ec/src/clj_kondo/impl/types.clj#L18-L51#2021-12-1216:10ikitommion master - adding type-hints for providers, starting with :map-of:
(require '[malli.provider :as mp])
(mp/provide
[^{::mp/hint :map-of}
{:a {:b 1, :c 2}
:b {:b 2, :c 1}
:c {:b 3}
:d nil}])
;[:map-of
; keyword?
; [:maybe [:map
; [:b int?]
; [:c {:optional true} int?]]]]#2021-12-1216:11ikitommithe current threshold with unified key & values schemas is 3, so:
(mp/provide
[{:a [1]
:b [1 2]
:c [1 2 3]}])
; [:map-of keyword? [:vector int?]]#2021-12-1216:12ikitommi… can be configured via options.#2021-12-1216:12ikitommiproviders start to be useful 🙂#2021-12-1220:50ikitommiadding :tuple inferreing, provider already 89 loc 🙀#2021-12-1220:52ikitommiwith :malli.provider/tuple-threshold defaulting to 3:
;; tuple-like with too few elements
[[:vector some?]
[[1 "2" true]
[2 "2" true]]]
;; tuple-like with enough samples
[[:tuple int? string? boolean?]
[[1 "2" true]
[2 "2" true]]
{::mp/tuple-threshold 2}]
;; tuple-like with enough samples
[[:tuple int? string? boolean?]
[[1 "2" true]
[2 "2" true]
[3 "3" true]]]
;; tuple-like with non-coherent data
[[:vector some?]
[[1 "2" true]
[2 "2" true]
[3 "3" "true"]]]
;; a homogenous hinted tuple
[[:tuple int? string? boolean?]
[^{::mp/hint :tuple} [1 "2" true]
[2 "2" true]]]
;; a hererogenous hinted tuple
[[:tuple int? string? some?]
[^{::mp/hint :tuple} [1 "2" true]
[2 "2" "true"]]]
;; invalid hinted tuple
[[:vector some?]
[^{::mp/hint :tuple} [1 "2" true]
[2 "2" true "invalid tuple"]]]#2021-12-1220:53ikitommihttps://github.com/metosin/malli/pull/593#2021-12-1621:45borkdude@ikitommi The newest clj-kondo (just released) supports automatically loading configs from .clj-kondo/*/*/config.edn - just saying :)#2021-12-1810:37ikitommithis is great news 🥳 !!#2021-12-1812:48ikitommi@U04V15CAJ so, this looks right? https://github.com/metosin/malli/pull/598#2021-12-1812:49ikitommifile-writing looks like:
#?(:clj
(defn save! [config]
(let [cfg-file (io/file ".clj-kondo" "configs" "malli" "config.edn")]
(io/make-parents cfg-file)
(spit cfg-file config)
config)))#2021-12-1812:50borkdudeit's recommended to write a directory that looks like:
<org>/<lib>/config.edn
#2021-12-1812:50borkdudeso then it would be:
<metosin>/<malli>/config.edn
#2021-12-1812:50borkdudebut this is usually for configs about malli and not provided by malli about other things#2021-12-1812:51borkdudeso maybe this would be better and would still give room for malli itself to export config for its own macros too:
<metosin>/<malli-type-anns>/config.edn
#2021-12-1812:52borkdudebut you are correct that the config isn't necessary anymore on the user's behalf#2021-12-1820:23ikitommifor some reason, the auto-loading doesn’t work for me :thinking_face:#2021-12-1820:23ikitommirepro: https://github.com/metosin/malli/pull/598#2021-12-1820:23borkdudehave you upgraded clj-kondo#2021-12-1820:24ikitommi➜ malli git:(clj-kondo-config-fix) ./bin/kaocha --focus malli.clj-kondo-test
[(.....)]
1 tests, 5 assertions, 0 failures.
➜ malli git:(clj-kondo-config-fix) cat .clj-kondo/configs/metosin/malli/config.edn
{:linters {:unresolved-symbol {:exclude [(malli.core/=>)]}, :type-mismatch {:namespaces {malli.clj-kondo-test {kikka {:arities {1 {:args [:int], :ret :int}, :varargs {:args [:int :int {:op :rest, :spec :int}], :ret :int, :min-arity 2}}}, siren {:arities {2 {:args [:ifn :coll], :ret :map}}}}}}}}%
➜ malli git:(clj-kondo-config-fix) clojure -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "2021.12.16"}}}' -M -m clj-kondo.main --lint test/malli/clj_kondo_test.cljc
linting took 163ms, errors: 0, warnings: 0
➜ malli git:(clj-kondo-config-fix) echo '{:config-paths ["configs/metosin/malli"]}' > .clj-kondo/config.edn
➜ malli git:(clj-kondo-config-fix) clojure -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "2021.12.16"}}}' -M -m clj-kondo.main --lint test/malli/clj_kondo_test.cljc
test/malli/clj_kondo_test.cljc:80:9: error: Expected: integer, received: string.
linting took 200ms, errors: 1, warnings: 0#2021-12-1820:24ikitommiI think that’s the latest one#2021-12-1820:25borkdudeyou should use the pattern .clj-kondo/*/*/config.edn , you have one dir too many#2021-12-1820:25ikitommioh, it was not **! 🙂#2021-12-1820:25borkdude.clj-kondo/metosin/malli#2021-12-1820:27borkdudebut I would rename this dir, as I said#2021-12-1820:27ikitommiworks like a charm, 🙇#2021-12-1820:27ikitommiyes, I will, just trying it out#2021-12-1820:27borkdude.clj-kondo/metosin/malli-types or so#2021-12-1820:27borkdudeso you can still export config for malli itself at one point#2021-12-1820:30ikitommiyes, changed that, will merge when tests pass. This is a really good change, makes the newbie user experience so much better.#2021-12-1820:30borkdudeI hope so.. :)#2021-12-1810:38ikitommimore batteries for value type inferring (https://github.com/metosin/malli/pull/597):
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])
(mp/provide
[{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"}
{:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"}])
; => [:map [:id string?]]
(mp/provide
[{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"}
{:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"}]
{::mp/value-providers {'string? {:uuid mt/-string->uuid}}})
; => [:map [:id :uuid]]
(mp/provide
[{"0423191a-5fee-11ec-bf63-0242ac130002" {:id "0423191a-5fee-11ec-bf63-0242ac130002"}
"09e59de6-5fee-11ec-bf63-0242ac130002" {:id "09e59de6-5fee-11ec-bf63-0242ac130002"}
"15511020-5fee-11ec-bf63-0242ac130002" {:id "15511020-5fee-11ec-bf63-0242ac130002"}}]
{::mp/value-providers {'string? {:uuid mt/-string->uuid}}})
; => [:map-of :uuid [:map [:id :uuid]]]#2021-12-1821:59respatializedI am encountering an arity error when I try to create a function schema for a multi-arity fn that accepts a vector of a specific type as an input.
(m/schema
[:function
[:=> [:cat [:schema [:* :int]]] :any]
[:=> [:cat [:schema [:* :int]] :boolean] :any]])
This produces the following error (I confirmed this with the latest tagged release of malli, 0.7.4):
{:type :malli.core/duplicate-arities,
:message :malli.core/duplicate-arities,
:data
{:infos
[{:min 0, :arity :varargs, :input [:cat [:schema [:* :int]]], :output :any}
{:min 1,
:arity :varargs,
:input [:cat [:schema [:* :int]] :boolean],
:output :any}]}}
I had figured that wrapping the input sequence schema in :schema would mark it as a distinct sequential value rather than a subcomponent of the input arg vector, but it appears that malli is treating an input vector of an arbitrary size as indicating that the function is supposed to have varargs, resulting in a clash between the arities of the function.#2021-12-1822:07respatializedrunning mg/generate on the arity input seqexes independently produces values that I expect to see:
(mg/generate [:cat [:schema [:cat [:* :int]]] :boolean])
;; => ((-2 2096836 3803 45 -49393) false)
(mg/generate [:cat [:schema [:cat [:* :int]]]])
;; => ((-1709 -2289 -38 3962880 -8 -948 13075997 77734 -16118422 635358 2 -1 -1))
#2021-12-1908:38ikitommiLooks like :schema has a bug related to m/-regex-min-max :
(m/-regex-min-max
(m/schema
[:cat [:schema [:* :int]]]))
; => {:min 0}
could you write an issue out of this?#2021-12-1908:39ikitommiwhile waiting, you can try:
(m/-regex-min-max
(m/schema
[:cat [:sequential :int]]))
; => {:min 1, :max 1}#2021-12-1908:41ikitommi.. and if it’s a vector, this is a way to describe it:
(m/schema
[:function
[:=> [:cat [:vector :int]] :any]
[:=> [:cat [:vector :int] :boolean] :any]])
ping @UFTRLDZEW#2021-12-1908:41ikitommibut, please issue the original, as it’s a bug.#2021-12-2015:11respatializedhttps://github.com/metosin/malli/issues/601 @U055NJ5CC thanks for taking a look.#2021-12-1822:07respatializedrunning mg/generate on the arity input seqexes independently produces values that I expect to see:
(mg/generate [:cat [:schema [:cat [:* :int]]] :boolean])
;; => ((-2 2096836 3803 45 -49393) false)
(mg/generate [:cat [:schema [:cat [:* :int]]]])
;; => ((-1709 -2289 -38 3962880 -8 -948 13075997 77734 -16118422 635358 2 -1 -1))
#2021-12-1906:38Nechemya Ungarjava.lang.RuntimeException: Unable to resolve symbol: qualified-keyword? in this context, compiling:(malli/core.cljc:167:51)
Hi all:
I have run into this error last week, at the beginning lein clean would resolve the issue but now our clojurescript project would not compile with malli .
If we add malli to our dependencies in project.clj and require malli.core the compiler spits out the aforementioned error. Is there a minimum clojure/clojurescript version?
this is our versions:
[org.clojure/clojurescript "1.10.439"]
[org.clojure/clojure "1.8.0"]
Thank you 🙏#2021-12-1908:44juhoteperiQualified-keyword? was added in 1.9#2021-12-1908:47Nechemya Ungarwow,
thank you for pointing it out (I am new to clojure)
so I wonder why it worked in the 1st place#2021-12-1909:06juhoteperiWell, you have the Cljs 1.10 so that does have qualified-keyword.
If you aren't using Malli in Clojure side, the problem could be macro use from Cljs and perhaps you changed something that caused Malli to be required in Cljs macro compilation time (so in Clj side)?#2021-12-1909:07juhoteperiAnd btw. Malli requires 1.10, there is another case why 1.9 isn't enough.#2021-12-1909:33juhoteperiReadme mentions 1.10 now#2021-12-1910:46Ben Slesstangential question, why were you using 1.8 to begin with?#2021-12-1910:53Nechemya UngarWell I came into the company half a year ago, I was doing typescript before... So I didn't touch anything related to the config#2021-12-1910:53Nechemya UngarThe code base is immense I thought using some schemas can help and malli was recommended to me over spec#2021-12-1909:23ikitommireleased [metosin/malli "0.7.5"], a fifth patch since 0.7.0, accumulated changes:
• https://github.com/clj-kondo/clj-kondo/blob/master/CHANGELOG.md#20211216 can load malli type configs automatically from new location (`.clj-kondo/metosin/malli-types/config.edn`), thanks @borkdude.
• use https://github.com/brandonbloom/fipp for fast pretty-printing the clj-kondo configs
• updated dependencies
• schema inferring supports value decoding via options
• :map-of inferring can be forced with `:malli.provider/hint :map-of` meta-data
• :tuple inferring (supports type-hints and threshold options)
• FIX Function with Sequential return value cannot define as function schema
• FIX `decimal?` predicate schema was removed in 0.7.0#2021-12-2202:34steveb8nQ: I have a typescript project which can expose react prop types. I’m wondering if there’s a way to convert these types to Malli using some kind of tooling? I don’t mind building the tool and submitting a PR if there is a way. Any typescript experts have ideas for this?#2021-12-2203:23steveb8nbackground: I have a contractor building react components in Storybook/typescript which I use in a re-frame app. This workflow is huge win#2021-12-2203:23steveb8nif I can figure out types -> Malli, I can enhance the testing story as well#2021-12-2205:19steveb8nI’m a typescript/babel noob so I need to figure out how to expose the types and then interop to generate Malli#2022-12-2716:59ikitommithere is https://github.com/tiagodalloca/malli-ts, is that still worked on @U4U6BDQTE ?#2022-03-0614:07Tiago Dall'Ocahey!#2022-03-0614:08Tiago Dall'OcaI need to show up here more often haha#2022-03-0614:08Tiago Dall'Ocayess, I'm working on it 🙂 (slowly)#2021-12-2214:20Ivan FedorovIs this normal that :e.order/shipping doesn’t resolve into a schema here? Or maybe I lost some understanding
(def registry:case
{:e.order/shipping
[:map [:address string?]]
:e/order
[:map
[:shipping-lines
[:vector [:ref :e.order/shipping]]]]})
(def schema:customer-order-case
(m/schema
[:schema
{:registry registry:case}
:e/order]))
(comment
(-> (m/deref-all schema:customer-order-case)
(m/children)
(last) (last)
(m/children) first ; -> [:ref :e.order/shipping]
(m/children) first type))
;; comment block evals to a keyword#2021-12-2214:57Ivan Fedorovok, I got it, I have to m/deref a [:ref :x] block, not :x itself#2021-12-2315:31Noah Bogart#re-frame has the function ->interceptor which has a parameter list of [& {:as m :keys [id before after]}] , which uses this new feature: https://clojure.org/news/2021/03/18/apis-serving-people-and-programs. how can i represent this in a malli function schema?#2021-12-2316:47Noah Bogarti have cobbled together this, but it feels pretty hacky, lol:
(m/=> ->interceptor
[:=>
[:cat
[:or
[:and
[:map
[:id qualified-keyword?]
[:before {:optional true} fn?]
[:after {:optional true} fn?]]
[:fn (fn [{:keys [before after]}] (or before after))]]
[:and [:catn
[:id [:cat [:= :id] qualified-keyword?]]
[:before [:? [:cat [:= :before] fn?]]]
[:after [:? [:cat [:= :after] fn?]]]]
[:fn (fn [[_ & args]] (pos? (count args)))]]]]
:any])
#2021-12-2316:48Noah Bogartis there a way to make generative testing work with defn schemas like this? (mi/check) reports there’s no generator attached to this schema#2021-12-2316:51Noah BogartExecution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.generator/no-generator {:options {:malli.core/function-checker #function[malli.generator/function-checker], :malli.generator/recursion {[:and [:map [:id qualified-keyword?] [:before {:optional true} fn?] [:after {:optional true} fn?]] [:fn #function[executor.interceptor/eval70188/fn--70190]]] 1, [:map [:id qualified-keyword?] [:before {:optional true} fn?] [:after {:optional true} fn?]] 1}}, :schema fn?}
#2022-12-2716:55ikitommi@nbtheduke bumped into that myself. Need to add proper support for the Keyword argument functions now also accept maps. Could you write an issue of that?#2022-12-2718:49Noah Bogarthttps://github.com/metosin/malli/issues/605#2022-12-2716:55Noah BogartYep, I can open an issue for it#2022-12-2717:00ikitommiis there a full spec for clojure destructuring?#2022-12-2717:09ikitommiseems to be that need to add something like :every and the merge of :map and :map-of to get proper support for parsing Clojure syntax.#2022-12-2717:10ikitomminaive 30min parser for extended Plumatic Schama defn syntax:
(m/parse
Bind
'[a :- :int
[b :- :int [c :- :string :as d :- [:vector :any]] & e :as f]
& {:keys [g h] :as i} :- [:map [:g :int] [:h :int]]])
;{:args [{:arg {:sym a}
; :- :-
; :schema :int}
; {:arg {:vec {:elems [{:arg {:sym b}
; :- :-
; :schema :int}
; {:arg {:vec {:elems [{:arg {:sym c}
; :- :-
; :schema :string}]
; :rest nil
; :as {:as :as
; :sym d
; :schema {:- :-
; :schema [:vector :any]}}}}}]
; :rest {:amp & :form {:arg {:sym e}}}
; :as {:as :as
; :sym f
; :schema nil}}}}]
; :rest {:amp &
; :arg {:arg {:map {:keys [g h] :as i}}
; :- :-
; :schema [:map [:g :int] [:h :int]]}}}#2022-12-2717:10ikitommi(def Bind
(m/schema
[:schema
{:registry
{"Schema" any?
"Amp" [:= '&]
"As" [:= :as]
"Local" [:and symbol? [:not "Amp"]]
"Map" [:map
[:keys {:optional true} [:vector ident?]]
[:strs {:optional true} [:vector ident?]]
[:syms {:optional true} [:vector ident?]]
[:or {:optional true} [:map-of simple-symbol? any?]]
[:as {:optional true} "Local"]
[:- {:optional true} "Schema"]]
"Vector" [:catn
[:elems [:* "SchematizedArgument"]]
[:rest [:? [:catn
[:amp "Amp"]
[:form "SchematizedArgument"]]]]
[:as [:? [:catn
[:as "As"]
[:sym "Local"]
[:schema [:? [:catn
[:- "Separator"]
[:schema "Schema"]]]]]]]]
"Argument" [:alt
[:catn [:sym "Local"]]
[:catn [:map "Map"]]
[:catn [:vec [:schema [:ref "Vector"]]]]]
"Separator" [:= :-]
"SchematizedArgument" [:alt
[:catn
[:arg "Argument"]]
[:catn
[:arg "Argument"]
[:- "Separator"]
[:schema "Schema"]]]
"Bind" [:catn
[:args [:* "SchematizedArgument"]]
[:rest [:? [:catn
[:amp "Amp"]
[:arg "SchematizedArgument"]]]]]}}
"Bind"]))#2022-12-2814:53ikitommiinferring function schemas, what would be better interpretation for these arguments:
'[a [b c]]
a) correct:
[:cat
:any
[:maybe
[:cat
[:? :any]
[:? :any]
[:* :any]]]]
b) strict:
[:cat
:any
[:schema
[:cat
:any
:any]]]
I would like to do b, but it’s Clojure and a is what the sequential destructuring does. let the user decide, default to a but allow a`strict` mode?#2022-12-2814:59ikitommi(defn kikka [a [b c]] [a b c])
(infer #'kikka)
;[:=>
; [:cat
; :any
; [:maybe
; [:cat
; [:? :any]
; [:? :any]
; [:* :any]]]]
; :any]
#2022-12-2814:59ikitommi(defn ^:malli/strict strict-kikka [a [b c]] [a b c])
(infer #'strict-kikka)
;[:=>
; [:cat
; :any
; [:schema
; [:cat
; :any
; :any]]]
; :any]
#2022-12-2815:01ikitommiwell, not doing the strict-mode now, but would be possible & easy#2022-12-2815:14Ben SlessWhat I'm missing most here is future support for named (or captured) type variables#2022-12-2816:55ikitommiwhat are named (or captured) type variables?#2022-12-2816:56ikitommi:cat could be :catn here, just need a good naming scheme for the unnamed things.#2022-12-2816:57ikitommialso, the whole thing is for full support of the plumatic syntax, so one can inline schemas.#2022-12-2816:58ikitommi(ms/defn kikka :- [:vector :int]
[a :- :int [b :- :int, c :- :int]]
[a b c])
#2022-12-2816:59Ben SlessThe type if a signature [a] which is any, but an association that constrains the captured name for a in the body of the expression. This lays the groundwork for type inference engines later. Then you can unify the knowledge about that association with the environment (call site) and body.
Essentially, introduce type variables / environment#2022-12-2817:00ikitommioh, that. I’m not doing that, hopefully smarter people will.#2022-12-2817:01Ben Sless😅#2022-12-2817:05Ben SlessI want to dive into it but I'm afraid I won't get anything#2022-12-2909:53ikitommiquestion: a new schema for "map or a sequence of map entries" or just an new property to :map schema? Relates to https://clojure.org/news/2021/03/18/apis-serving-people-and-programs.#2022-12-2909:55ikitommia) (m/validate [:map-like [:a :int]] [[:a 1]]) ;=> true
b) (m/validate [:map {:coerce true}: [:a :int]] [[:a 1]]) ;=> true#2022-12-2909:57ikitommiI'm currently thinking of going with a, as this is a special case. Both ways, it's just few lines of extra code I think#2022-12-2909:58ikitommialso, what would be a good name for a new schema type, :map-like ? :every ?#2022-12-2909:59ikitommido you @nbtheduke the opinion for this?#2022-12-2909:59Ben Sless:entries#2022-12-2910:00Ben SlessThen :map is :entries + map?#2022-12-2910:03ikitommispec seems to have s/keys* for this#2022-12-2910:48ikitommithing is, transformers, parsers and explainers need separate code if we want to retain the original entry sequence. forcing the data to be a map allows us to reuse the current code.#2022-12-2910:49ikitommifor the Keyword argument functions now also accept maps thing, we just need a map, so a simple new way to coerce the entry sequence into a map would do.#2022-12-2913:04Noah BogartOne thing to note, the input isn't a sequence of map entries, but a sequence of alternating keys and values, which can be in any order #2022-12-2913:29ikitommiyes. I guess this is correct:
(require '[malli.destructure :as md])
(require '[malli.generator :as mg])
(defmethod mg/-schema-generator :any [_ _] (mg/generator :string))
(-> '[& {:as m :keys [id before after]}]
(md/parse)
:schema
(mg/sample {:size 10, :seed 42}))
;(({:after ""})
; (:id "F")
; ()
; ({:after ""})
; (:before "n" :after "ai2" :id "5FI0" :id "")
; ({:after "qP3t"})
; ()
; ({:id "j"})
; ()
; (:before
; "2"
; :after
; "HW5Mn3"
; :after
; "gv9m93"
; :before
; "GVDI2b"
; :before
; "fhR"
; :before
; "F8562"
; :after
; "lS"
; :before
; "Z7y0nz"
; :before
; "7G"))#2022-12-2913:31ikitommiWIP https://github.com/metosin/malli/pull/606#2022-12-2915:23Noah Bogartthis matches the implementation, as far as i can tell! very cool#2022-12-2915:32Noah Bogartlooks like you’re missing this from the test suite, i’m not sure if you’re missing it from the schema:
user=> (defn example [{:keys [id before after] :as m}] [id before after m])
#'user/example
user=> (example '(:id 1 :before 2 :after 3 :missing 4))
[1 2 3 {:id 1, :before 2, :after 3, :missing 4}]#2022-12-2915:34Noah Bogartwhich is to say, the bind is a destructured map, but the input can be a list of key-value pairs#2022-12-2915:36Noah Bogartweird thing, it can’t be a vector:
user=> (example [:id 1 :before 2 :after 3 :missing 4])
[nil nil nil [:id 1 :before 2 :after 3 :missing 4]]
#2022-12-2915:36ikitommiyou can pass in it as a list 🤯 by default???#2022-12-2915:38Noah Bogarthah yeah, it’s not really talked about anywhere, but the current version of clojure.core/destructure allows it: https://github.com/clojure/clojure/blob/clojure-1.11.0-alpha3/src/clj/clojure/core.clj#L4434-L4439#2022-12-2918:21ikitommi@nbtheduke, added support for it too. starts to smell like a new :map-destructuring Schema, which would hide the sequential part. Ugly, but works(?):
(-> '[a {:keys [b c]
:strs [d e]
:syms [f g]
:or {b 0, d 0, f 0} :as map}]
(md/parse)
:schema)
;[:cat
; :any
; [:altn
; [:map [:map
; [:b {:optional true} :any]
; [:c {:optional true} :any]
; ["d" {:optional true} :any]
; ["e" {:optional true} :any]
; ['f {:optional true} :any]
; ['g {:optional true} :any]]]
; [:args [:schema
; [:*
; [:alt
; [:cat [:= :b] :any]
; [:cat [:= :c] :any]
; [:cat [:= "d"] :any]
; [:cat [:= "e"] :any]
; [:cat [:= 'f] :any]
; [:cat [:= 'g] :any]
; [:cat :any :any]]]]]]]#2022-12-2918:21Noah Bogartamazing. thanks so much for tackling this. destructuring in clojure is really weird haha#2022-12-2918:24ikitommi(def Schema
(-> '[a {:keys [b c]
:strs [d e]
:syms [f g]
:or {b 0, d 0, f 0} :as map}]
(md/parse)
:schema))
(m/parse Schema [1 {:b 1, 'f 3, "e" 2, :extra 42}])
; => [1 [:map {:b 1, f 3, "e" 2, :extra 42}]]
(m/parse Schema [1 '(:c 1, , f 3, "e" 2, :extra 42)])
; => [1 [:args [[:c 1] [f 3] ["e" 2] [:extra 42]]]]#2022-12-2914:49ikitommia thought experiment, should we have more argument relationship markers, e.g. :- (is a) and :< (a subset of)? could also go EXTREME EVIL and introduce real math symbols like :⊂ 👿
(def User
[:map
[:id :uuid]
[:name :string]
[:age :int]])
;; argument is exactly User
[{:keys [id age]} :- User]
; => [:cat [:map [:id :uuid] [:name :string] [:age :int]]]
;; argument should be subset of user (mark others as optional)
[{:keys [id age]} :< User]
; => [:cat [:map [:id :uuid] [:name {:optional true} :string] [:age :int]]]
#2022-12-2914:51ikitommialso, would be awesome if tools like #cursive and #calva would have special markers for the type hints, e.g. dim them out so it’s easier to read.#2022-12-2914:54ikitommi… for fully qualified keys, the key definitions could be pulled from the registry:
(mm/def ::id :uuid)
(mm/def ::name :string)
(mm/def ::age :int)
;; argument is exactly User
[{::keys [id age]}]
; => [:cat [:map ::id ::age]]#2022-12-2914:54dharriganWe use (still 😞 ) compojure-sweet for some of our APIs, and I see a lot of :- in the path-params, body.... etc...and well, personally, I found it hard to know what :- meant etc...#2022-12-2914:56ikitommiin compojure-api, there is a lot of extra syntax, also the fnk syntax. sorry for all that 🙂#2022-12-2914:56dharriganI have to deal with it every day 😞 I definitely see lessons where learnt with the new improved reitit library 🙂#2022-12-2914:57dharriganSo, I'm not for, nor against, additional markers, I'm rather on the fence (just more stuff to learn I suppose!)#2022-12-2914:59ikitommiyeah, any support for inline typehints is a compromise. If IDEs would support that properly and there would be just one way to doing those, would be great.#2022-12-2915:19Noah Bogarti think i’d prefer words instead of single characters for such things. :exact or :exactly or :is-a are easier to parse than :- in my opinion. subset feels like Typescript-style structural typing, which clojure supports out of the box with open maps#2022-12-2915:20dharriganI'm with Noah on this one too, I would have to translate :- into is-a in my head as well. Nothing wrong in being wordy, when it comes to comprehension.#2022-12-2915:20dharriganmy 2c`s 🪙#2022-12-2915:25ikitommi#2022-12-2915:26ikitommi^:-- that’s what I would love to have, to remove the type/schema clutter if used.#2022-12-2915:30Noah Bogartah, i understand now that :- is supposed to be like static type declarations, that makes more sense#2022-12-2916:19Yehonathan SharvitIs there a commonly agreed way to document the meaning of each value in a :enum?#2022-12-2918:18ikitommi@viebel don’t think there is. ideas welcome#2022-12-2918:58Ben Slessenumn
Where the keys are the enumeration and values are properties#2022-12-3007:47Yehonathan SharvitI don't get what you mean @UK0810AQ2#2022-12-3007:55Ben SlessLike the map syntax, where the keys are the enumeration and optionally you could provide a properties map#2022-12-3008:47ikitommi:enumn (or similar) would be good for key->value pair mappings.#2022-12-3008:48ikitommi(def MyEnum
[:enumn
[:small {:description "so small"} "small"]
[:medium {:description "such medium"} "medium"]])#2022-12-3008:49ikitommi(m/validate MyEnum "small") ; => true
#2022-12-3008:50ikitommi(m/parse MyEnum "small") ; => [:small "small"]
#2022-12-3008:50ikitommi:thinking_face:#2022-12-3008:50ikitommiawesome#2022-12-3008:51ikitommie.g. :cat + :catn, :or + : orn, …#2022-01-0208:18Yehonathan SharvitI like the idea of enumn .
I am wondering if there is any meaning to the keys (e.g. :small and :medium ).#2022-01-0208:27Ben SlessI was thinking of a slightly different semantic, [:enumn [v1 {:doc "foo}] [v2] v3], where the keys are the enumerations, and the syntax can be value | [value ?properties]#2022-01-0505:34Yehonathan SharvitI prefer what you suggested @UK0810AQ2 as we don't have to duplicate the enum values#2022-01-1009:35Yehonathan SharvitOpened an issue about this feature request https://github.com/metosin/malli/issues/613#2022-12-3008:53ikitommiOne of the oldest (and annoying) issue is how to describe a map + map-of. Need to take a stab at it, here are the options:
1) ternary closed
[:map {:closed [:string :int]}
[:x :int]
[:y :int]]
2) new extra-keys (or such)
[:map {:extra-keys [:string :int]}
[:x :int]
[:y :int]]
3) ::m/default (like in :multi)
[:map
[:x :int]
[:y :int]
[::m/default [:map-of :string :int]]]
#2022-12-3008:56ikitommileaning on 3, because:
• it’s coherent way to describe “default in case none of the defined keys matched”
• it’s easy to remove or add the key, e.g. (mu/assoc MyMap ::m/default [:map-of :uuid MyMap])
• :map-of already supports key-decoding so things like :uuid keys just work oob#2022-12-3008:56ikitommicomments welcome, original issue here: https://github.com/metosin/malli/issues/43#2022-12-3009:58Ben Sless4: [:map ^:of [int? int?]]#2022-12-3011:16ikitommimetadata looks good when writing, not that much when reading / serializing.#2022-12-3011:22Ben SlessI think this example is confusing because the map of spec and the entries don't match.
A property of of on the map makes the most sense imo#2022-12-3011:36ikitommi:of sounds ok. what do you mean by:
> the map of spec and the entries don’t match.#2022-12-3011:38ikitommihow would a schema in the properties be reported in m/explain?#2022-12-3011:41Ben SlessIf its :string then it can't have a keyword key. It has to be a union and not a superset. Which is confusing#2022-12-3011:43Ben SlessBut I'm not sure map-of and map should be unified#2022-12-3011:47ikitommiI read it “it’s a map with keyword keys :x and :y, the rest of the keys should be :string -> :int. Same with plumatic:
{:x s/Int, :y s/Int, s/Str s/Int}#2022-12-3011:48ikitommie.g. {:x 1, :y 2, "z" 3, "å" 4} is valid in it.#2022-12-3011:49Ben SlessAs a user who has to work and communicate with others via code, I wouldn't want this to be valid#2022-12-3011:50Ben SlessI'd prefer that the specific keys will be a subset of the key schema and values be a subset of the value schema, not that the entire map describe a union#2022-12-3011:51ikitommiI see you point, would not want to use that myself, but that’s what JSON Schema, Plumatic and many others have atm.#2022-12-3011:52ikitommifor the destructuring, I would like to describe the namespaced keys with that#2022-12-3011:55Ben SlessJson schema has no notion of keywords, though#2022-12-3011:58Ben SlessAnd I don't mind saying Plumatic made a mistake by allowing it#2022-12-3011:50ikitommi5) wrap the extra keys with something like :schema to mark it’s a schema, not a real key.
[:map
[:x :int]
[:y :int]
[[:schema :string] :int]]
#2022-12-3011:51Ben SlessWouldn't it require a special case in parsing?#2022-12-3011:52ikitommiyes#2022-12-3011:54Ben SlessLeaving surprises to your future self?#2022-12-3011:56ikitommiyes, I don’t think there is a right answer to this, just compromises. Not a fan of those (the reason the issue has been open for so long)#2022-12-3011:52Ben SlessShould map-of and map schemas be unified? One describes nominal tuples, the other a set of tuples#2022-12-3011:53ikitommihow could/should they be unified?#2022-12-3011:53juhoteperi::m/default makes sense, but using :map-of together with that seems funny. In :map schema the items are key-value pairs, :map-of describes full map.
Something like [::m/default [:map-entry :string :int]] or just [::m/default :string :int] or [::m/default [:string :int]] (just force the :map default entry to always have two items) could be cleaner.#2022-12-3011:55ikitommiproblem with [::m/default :string :int] is that when you ask for m/children of the map, you get funny results.#2022-12-3011:56juhoteperiMaybe the :map-of makes sense as there can be multiple "default" / extra keys#2022-12-3011:57juhoteperiMaybe using ::m/extra name would describe better that this is the schema for those keys that aren't directly defined in :map schema#2022-12-3011:58ikitommiIn JSON Schema -land:
{
"type": "object",
"properties": {
"number": { "type": "number" },
"street_name": { "type": "string" },
"street_type": { "enum": ["Street", "Avenue", "Boulevard"] }
},
"additionalProperties": { "type": "string" }
}#2022-12-3012:03Ben SlessIt reads to me like the specified properties and additional properties should have the same key type#2022-12-3012:03Ben SlessBut json is limited#2022-12-3012:08ikitommiyes, we do not have that limitation#2022-12-3012:09ikitommibut, the extra keys are just for the case none of the actual keys hit#2022-12-3012:09ikitommithere is also :patternedProperties and other silly things.#2022-12-3012:27Ben Sless😕#2022-12-3012:09ikitommithe root need for this NOW btw is how to describe the namespaced keys in the destructuring syntax, elegantly. e.g. what is the schema for the map here:
(ns demo)
(let [{:keys [a1] ::keys [a2], :kikka/keys [a3]
:syms [b1] ::syms [b2] :kikka/syms [b3]
:strs [c1]
:or {a1 0} :as map}
{:a1 1, ::a2 2, :kikka/a3 3
'b1 4 'demo/b2 5, 'kikka/b3 6
"c1" 5}]
[a1 a2 a3 b1 b2 b3 c1])
; => [1 2 3 4 5 6 5]#2022-12-3012:11ikitommithought that would be a good reason to add the “extra keys” here, but not sure if that is needed. could be just :multi with dispatch on key qualification :thinking_face:#2022-12-3012:43Ben SlessOr map of enum to something?#2022-12-3012:45Ben SlessThis brings me back to my question about a schema about entries#2022-12-3012:53Ben Slessentry := qualified | simple | or | as
qualified := [ns/kind form]
simple := [kind form]
kind := keys | syms | strs
or := [:or map-of,,,]
as := [:as symbol]
#2022-01-0116:54ikitommiunexpected help for the existing stuff:
(defn -map-like [x]
(or (map? x)
(and (seqable? x)
(every? (fn [e] (and (vector? e) (= 2 (count e)))) x))))
(defn -keys-syms-key [k]
(-> k name #{"keys" "syms"}))
(def MapLike
(m/-collection-schema
{:type :map-like
:empty {}
:pred -map-like}))
(m/parse
[MapLike
[:or
[:tuple [:= :keys] [:vector ident?]]
[:tuple [:= :strs] [:vector ident?]]
[:tuple [:= :syms] [:vector ident?]]
[:tuple [:= :or] [:map-of simple-symbol? any?]]
[:tuple [:= :as] symbol?]
[:tuple [:and :qualified-keyword [:fn -keys-syms-key]] [:vector ident?]]]]
'{:keys [b]
:strs [c]
:syms [d]
:demo/keys [e]
:demo/syms [f]
:or {b 0, d 0, f 0} :as map})
;{:keys [b]
; :strs [c]
; :syms [d]
; :demo/keys [e]
; :demo/syms [f]
; :or {b 0, d 0, f 0} :as map}
… not the most performant, but good for the destructuring case 🥳#2022-01-0116:54ikitommiunexpected help for the existing stuff:
(defn -map-like [x]
(or (map? x)
(and (seqable? x)
(every? (fn [e] (and (vector? e) (= 2 (count e)))) x))))
(defn -keys-syms-key [k]
(-> k name #{"keys" "syms"}))
(def MapLike
(m/-collection-schema
{:type :map-like
:empty {}
:pred -map-like}))
(m/parse
[MapLike
[:or
[:tuple [:= :keys] [:vector ident?]]
[:tuple [:= :strs] [:vector ident?]]
[:tuple [:= :syms] [:vector ident?]]
[:tuple [:= :or] [:map-of simple-symbol? any?]]
[:tuple [:= :as] symbol?]
[:tuple [:and :qualified-keyword [:fn -keys-syms-key]] [:vector ident?]]]]
'{:keys [b]
:strs [c]
:syms [d]
:demo/keys [e]
:demo/syms [f]
:or {b 0, d 0, f 0} :as map})
;{:keys [b]
; :strs [c]
; :syms [d]
; :demo/keys [e]
; :demo/syms [f]
; :or {b 0, d 0, f 0} :as map}
… not the most performant, but good for the destructuring case 🥳#2022-12-3104:17SwapneilHow does one make a relation between the input and output of a function? For instance, if I had the function (defn dothings [x y] (/ x y)), I want to check that (zero? (mod x :ret)). In spec you can do this with the :fn option in fdef, but I can't find any way to use malli's :fn option this way in a :=> schema#2022-12-3107:45ikitommi@ss2961 currently, no. but would be easy to add. Please write an issue out of that.#2022-12-3107:47ikitommie.g. with map-syntax:
{:type :=>
:input {:type :cat, :children [{:type :int}]}
:output :int}
=>
{:type :=>
:input {:type :cat, :children [{:type :int}]}
:output :int
:fn ...}
not sure where that should be in the vector-syntax. optional third element?#2022-12-3107:57Ben SlessProperties?#2022-12-3108:16ikitommiIn this case, I think it's the way to go. The :=>satisfies the SchemaAST so it can read & write them from there. Brilliant!#2022-12-3107:54ikitommiadded some options to destructuring parser:
(defn parse
"Takes a destructuring bindings vector (arglist)
and returns a map with keys:
| key | description |
| ---------------|-------------|
| `:raw-arglist` | the original arglist (can have type-hints)
| `:arglist` | simplified clojure arglist (no type-hints)
| `:schema` | extracted malli schema
| `:parsed` | full parse results
Parsing can be configured using the following options:
| key | description |
| -----------------------|-------------|
| `::md/inline-schemas` | support plumatic-style inline schemas (true)
| `::md/sequential-maps` | support sequential maps in non-rest position (true)
| `::md/required-keys` | are destructured keys required (false)
| `::md/closed-maps` | are destructured maps closed (false)
| `::md/references` | are schema references used (false)
Examples:
(require '[malli.destructure :as md])
(-> '[a b & cs] (md/parse) :schema)
; => [:cat :any :any [:* :any]]
(-> '[a :- :string, b & cs :- [:* :int]] (md/parse) :schema)
; => [:cat :string :any [:* :int]]"#2022-12-3107:55ikitommiany comments to https://github.com/metosin/malli/pull/606? naming, defaults etc. will ship that… next year 🙂#2022-01-0215:01ikitommiUpdating the README: pulling out malli function schemas from normal clojure functions:
(require '[malli.destructure :as md])
(def infer (comp :schema md/parse))
(defn kikka
([a] [a])
([a b & cs] [a b cs]))
(->> #'kikka
meta
:arglists
(map infer)
(map (fn [s] [:=> s :any]))
(into [:function]))
;[:function
; [:=> [:cat :any] :any]
; [:=> [:cat :any :any [:* :any]]#2022-01-0215:05ikitommicould add a helper to enable that with malli.dev so that one could auto-infer all/interesting Vars at dev-team to get pretty runtime errors + clj-kondo mappings for free too.#2022-01-0218:30ikitommimerged. Before jumping into the-next-thing-I-need, I’ll try to address all PRs. Really good stuff there 🙇#2022-01-0218:32Ben SlessRegarding the string char ranges, @U02AH3D0HEV had a great idea for putting it in a lookup array, great perf#2022-01-0218:35Ben SlessAlso experimented with emitting a datomic(like) schema from schema#2022-01-0319:44SwapneilHow do you make a spec for a :repeat or :sequential with a certain length? I tried adding :count and :length in the properties, but neither seems to have an effect.
My current spec is [:repeat {:min 0 :max 9} :int]]#2022-01-0414:32ikitommitry:
(m/explain [:repeat {:min 2, :max 2} :int] [1 2])
; nil
(m/explain [:repeat {:min 2, :max 2} :int] [1])
;{:schema [:repeat {:min 2, :max 2} :int],
; :value [1],
; :errors ({:path [0]
; :in [1]
; :schema :int
; :value nil
; :type :malli.core/end-of-input})}#2022-01-0414:23pinkfrogIs the map syntax the recommended way to go? https://github.com/metosin/malli#map-syntax#2022-01-0414:33ikitommivector-syntax is the way to go, but map should work too.
> NOTE: Map Syntax / SchemaAST is considered as alpha and subject to change.#2022-01-0414:34ikitommi#2022-01-0414:35ikitommiwith cursive + Clojure Extras + clj-kondo. Looking good 🙂#2022-01-0414:37dharriganI am liking the metadata version#2022-01-0414:37dharriganto me, precisely what metadata is all about 🙂#2022-01-0415:13emccuei like it too, just not that dev tooling can’t pick it up without a manual refresh#2022-01-0415:15emccueand the m/=> version there doesn’t work because if you load the whole file you load the schema, which triggers wrapping the function, and then load the function which will be unwrapped#2022-01-0415:31Karol WójcikPlumatic style. Cant wait to see it released ;3#2022-01-0415:54ikitommiI think we can solve the dev-tooöing issues with first two, with polling and var-watching.#2022-01-0416:31Michael Gardnerany progress on the timing problem with metadata?
https://clojurians.slack.com/archives/C03S1KBA2/p1628602192164200#2022-01-0416:41ikitommiI don't think so, but adding a queue with a small delay should do the trick here.#2022-01-0417:42Michael Gardnerseems tricky to deal with the case where a var is defined and then immediately used#2022-01-0418:18Ben Slessteaser - how does emitting malli schema fron jackson annotations sound? 😛#2022-01-0420:55emccuei swear i dislike actual physical people named jackson at this point just by association#2022-01-0507:22Ben SlessI always think of Daniel Jackson from Stargate which puts me in a more positive mindset#2022-01-0507:55Ben SlessFiled under "thanks I hate it"
https://github.com/bsless/malli-jackson#2022-01-0509:48eskosCan/could malli be used to extract/subselect a structure from a larger data structure?
What I’m trying to achieve is that my CLJS app has one huge app-db ratom and from that I’d like to subselect certain parts which gets stored persistently with as little pain and whacking as possible, and I sort of had a thought that hey, it’d be nice to define the subselect as malli schema so that I could validate that the subselect is even persistable, plus malli schema is way easier to document than whatever else I’ve been thinking as solution to this so far. From system integrity’s point of view this could also work quite nicely, as marking specific branches/leaves of the data structure as persistable would mean writing a proper validator for it -> I wouldn’t have to worry someone breaking the, uh, persistability. And of course this is pretty straightforward thing to document to team members, both current and those who’ll join the project later that hey, just update the persistent malli schema if your data needs to be persisted.#2022-01-0510:13ikitommiThere is mt/strip-extra-keys-transformer to "select" a sub-schema.#2022-01-0513:49eskosHmm, interesting. :thinking_face:#2022-01-0512:59lauriThe following code snippet works with malli version 0.6.2:
(def instant
(m/-simple-schema
{:type 'instant
:pred (partial instance? Instant)
:type-properties {:error/message "should be instant"}}))
(def Foo
[:schema
{:registry
{::xyz
[:map [:baz instant]]}}
::xyz])
(mu/merge [:map [:foo string?]]
[:map [:bar Bar]])
=>
[:map
[:foo
[:schema
{:registry #:test-namespace{:xyz [:map [:baz instant]]}}
:test-namespace/xyz]]
[:bar string?]]
but fails since version 0.7.0 with:
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.core/invalid-schema {:schema instant}
How can i execute it with the newest version on malli?#2022-01-0513:05lauriwhen the type of the :baz field is changed to string? then it also works with the latest version but i’d like to be able to use a custom type..#2022-01-0513:57lauriseems that the malli.core/schema function fails in the newest version:
(m/schema Foo)
Error printing return value (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.core/invalid-schema {:schema instant}#2022-01-0610:21ikitommi@U082J3TME It was a regression that is should be fixed with https://github.com/metosin/malli/commit/abef62dc682af7e1f61b841e5fed4038a1ccb01c, not released yet#2022-01-0610:21ikitommicould you retest with latest code in master?#2022-01-0610:22ikitommie.g. property registries were written as forms, which was a bad idea as there is no way to re-construct non-registered schemas using just the m/type information.#2022-01-0612:14laurijust checked with master and it works well now. Thanks a lot!#2022-01-0520:53annapawlickaHey folks, I’m trying to figure out if there’s a way to merge fields inside of registry. I’ve tried utility functions, tried declarative :merge , they don’t work. Here’s a small example of what I’m trying to do:
(def schema-registry-sample
{::payment-authorization [:map
[:id :string]
[:approvalNumber :string]
[:authResultDescription :string]
[:authorizationDate :string]]
::sales-order-line-item [:map
[:id :string]
[:authorization [:maybe
[:merge
::payment-authorization ;; <<<< I want to merge this with the fields below
[:map
[:type :string]
[:activationCode :string]
[:productCode :string]]]]]
[:offer {:optional true} [:multi {:dispatch :offerProgram}
["INSTANTSAVINGS" [:merge
::offer ;; <<<< I want to add required field based on a certain condition
[:map [:gtin :string]]]]
[::m/default ::offer]]]]
:sales-order [:map
[:paymentAuthorization ::payment-authorization]
[:salesOrderLineItem [:vector ::sales-order-line-item]]]})
(def sales-order-explainer-sample
(m/explainer [:schema {:registry schema-registry-sample}
::sales-order]))
The example here doesn’t work
{:type clojure.lang.ExceptionInfo
:message ":malli.core/invalid-schema {:schema :merge}"
:data {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :merge}}
:at [malli.core$_fail_BANG_ invokeStatic "core.cljc" 137]}
Other merge approaches fail as well. Any ideas? Is this just not doable right now?#2022-01-0616:30ikitommihi @U05087MJH! the :merge is not enabled by default. You can register it globally with:
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas) ;; malli defaults
(mu/schemas))) ;; plus the utility schemas
#2022-01-0616:31ikitommiso, this works:
(ns demo
(:require [malli.registry :as mr]
[malli.core :as m]
[malli.util :as mu]))
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas) ;; malli defaults
(mu/schemas))) ;; plus the utility schemas
(def schema-registry-sample
{::payment-authorization [:map
[:id :string]
[:approvalNumber :string]
[:authResultDescription :string]
[:authorizationDate :string]]
::sales-order-line-item [:map
[:id :string]
[:authorization [:maybe
[:merge
::payment-authorization ;; <<<< I want to merge this with the fields below
[:map
[:type :string]
[:activationCode :string]
[:productCode :string]]]]]
[:offer {:optional true} [:multi {:dispatch :offerProgram}
["INSTANTSAVINGS" [:merge
::offer [:map [:gtin :string]]]]
[::m/default ::offer]]]]
::offer [:map]
::sales-order [:map
[:paymentAuthorization ::payment-authorization]
[:salesOrderLineItem [:vector ::sales-order-line-item]]]})
(def sales-order-explainer-sample
(m/explainer [:schema {:registry schema-registry-sample}
::sales-order]))#2022-01-0616:35ikitommiIf you want spec-like custom mutable registry, here’s the thing:
(ns demo
(:require [malli.registry :as mr]
[malli.core :as m]
[malli.util :as mu]))
(def registry* (atom {}))
(defn register! [type ?schema]
(swap! registry* assoc type ?schema))
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
(mu/schemas)
(mr/mutable-registry registry*)))
(register!
::payment-authorization
[:map
[:id :string]
[:approvalNumber :string]
[:authResultDescription :string]
[:authorizationDate :string]])
(register!
::sales-order-line-item
[:map
[:id :string]
[:authorization [:maybe
[:merge
::payment-authorization
[:map
[:type :string]
[:activationCode :string]
[:productCode :string]]]]]
[:offer {:optional true} [:multi {:dispatch :offerProgram}
["INSTANTSAVINGS" [:merge
::offer
[:map [:gtin :string]]]]
[::m/default ::offer]]]])
(register!
::offer
[:map])
(register!
::sales-order
[:map
[:paymentAuthorization ::payment-authorization]
[:salesOrderLineItem [:vector ::sales-order-line-item]]])
(def sales-order-explainer-sample
(m/explainer ::sales-order))#2022-01-0621:07annapawlickaThank you @U055NJ5CC! I just realized I posted incomplete example, but you made it work anyway 🙂#2022-01-0720:49annapawlicka@U055NJ5CC Any chance to merge the mu/schemasinto my schema? I’d like to make the registry in this ns immutable as we’ll be using a lot of different schemas and there’s always a risk some other ns will set the global registry to something else. Simple merging doesn’t seem to work.#2022-01-0820:19ikitommiI think it’s possible. Need few minutest to test that one out.#2022-01-0820:26ikitommihere’s one way to do it: https://gist.github.com/ikitommi/af280065df4cb16a17123558e41463d3#2022-01-0820:27ikitommie.g. create a common registry for the base-schemas (immutable), pass it as argument to m/schema , can still add local schemas on top of it.#2022-01-0820:28ikitommimost workers (validators, explainers, parsers, unparsers, generators) are cached to the instance, so first m/validate on the Schema instance will create the validator, next will use the cached one.#2022-01-0820:37ikitommihappy to give a tech talk / discussion about malli for you team you are interested. There are lot’s of options of doing things. Also curious on where are you using it.#2022-01-0908:49ikitommitwo more options (added to gist):
;;
;; 3) registering utility schema via local registry (requires latest code from MASTER)
;;
(def schema-registry-sample
{:merge (mu/-merge)
::payment-authorization [:map
[:id :string]
[:approvalNumber :string]
[:authResultDescription :string]
[:authorizationDate :string]]
::sales-order-line-item [:map
[:id :string]
[:authorization [:maybe
[:merge
::payment-authorization ;; <<<< I want to merge this with the fields below
[:map
[:type :string]
[:activationCode :string]
[:productCode :string]]]]]
[:offer {:optional true} [:multi {:dispatch :offerProgram}
["INSTANTSAVINGS" [:merge
::offer [:map [:gtin :string]]]]
[::m/default ::offer]]]]
::offer [:map]
::sales-order [:map
[:paymentAuthorization ::payment-authorization]
[:salesOrderLineItem [:vector ::sales-order-line-item]]]})
(def sales-order
(m/schema
[:schema {:registry schema-registry-sample}
::sales-order]))#2022-01-0908:49ikitommi;;
;; 4) using local component directly
;;
(def Merge (mu/-merge))
(def schema-registry-sample
{::payment-authorization [:map
[:id :string]
[:approvalNumber :string]
[:authResultDescription :string]
[:authorizationDate :string]]
::sales-order-line-item [:map
[:id :string]
[:authorization [:maybe
[Merge
::payment-authorization ;; <<<< I want to merge this with the fields below
[:map
[:type :string]
[:activationCode :string]
[:productCode :string]]]]]
[:offer {:optional true} [:multi {:dispatch :offerProgram}
["INSTANTSAVINGS" [Merge
::offer [:map [:gtin :string]]]]
[::m/default ::offer]]]]
::offer [:map]
::sales-order [:map
[:paymentAuthorization ::payment-authorization]
[:salesOrderLineItem [:vector ::sales-order-line-item]]]})
(def sales-order
(m/schema
[:schema {:registry schema-registry-sample}
::sales-order]))#2022-01-1020:57annapawlickaThank you! It looks like there’s quite a few options. We might take you up on your tech talk offer but we need to gather a list of questions first 🙂
We’ve been slowly replacing Plumatic schema with malli. Plus we started adding malli to a recent project that had no Plumatic schema - we use it to validate payload from services we call, but also to validate data that we send in our requests to them. Generation of test data is there as well but I think validation is the biggest part. And since these are all different data sources that we validate, we’re trying to find the best way to organize it. We’ve just started with it in December IIRC, so it will possibly grow. It also helped us to identify how much of the payload we actually use (it’s a big project, different graphql resolvers using the same source of data in different ways), and helped us make a decision around choosing the right response version.
@UEH4D93GS has been looking into it from a different project than I so he might have a different perspective.#2022-01-0619:17Ben SlessAfter opening this I noticed there's been a similar request in this channel in the past. Anyone did work on the subject?
https://github.com/metosin/malli/issues/609#2022-01-0713:44PanelTrying to understand why transformers won't compose, in this ex, (mt/transformers {:decoders ...}) is compiled but not run when the second transformer (mt/key-transformer ...) is present.
If I remove (mt/key-transformer ...) then (mt/transformers {:decoders ...}) run just fine.
I'm guessing this is related to interceptor :before and :after clashing for some reason.
(ma/decode
[:map ["ContactMethodCde"
[:enum {:lkp {"pcme" "Email"
"pcmp" "Phone"
"pcmm" "Mail"}}
"Mail" "Phone" "Email"]]]
{"ContactMethodCde" "pcme"}
(mt/transformer
{:decoders {:enum {:compile (fn [schema _]
(let [m (:lkp (ma/properties schema))]
(fn [x]
(prn x)
(m x))))}}}
(mt/key-transformer
{:decode {"ContactMethodCde" :contact-method}})))
This work, notice the :leave interceptor on :map decoders.
(ma/decode
[:map ["ContactMethodCde"
[:enum {:lkp {"pcme" "Email"
"pcmp" "Phone"
"pcmm" "Mail"}}
"Mail" "Phone" "Email"]]]
{"ContactMethodCde" "pcme"}
(mt/transformer
{:decoders
{:enum {:compile (fn [schema _]
(let [m (:lkp (ma/properties schema))]
{:enter(fn [x]
(prn x)
(m x))}))}
:map {:leave (mt/-transform-map-keys {"ContactMethodCde" :contact-method})}}}))
It wont work when the decoder is set to :enter :map {:enter ...}#2022-01-0715:54Grant HornerIs there an equivalent to spec-alpha2's select in malli? i.e., it takes a :map schema and makes everything optional other than the specified keys?#2022-01-0721:41Panelhttps://cljdoc.org/d/metosin/malli/0.7.5/api/malli.util#required-keys maybe ?#2022-01-0820:17ikitommisomething like this?
(require '[malli.core :as m])
(require '[malli.util :as mu])
(defn select [schema keys]
(-> schema
(mu/optional-keys)
(mu/required-keys keys)))
(select
[:schema {:registry {::name :string
::title [:enum "boss" "not-boss"]
::age :int
::skills [:set [:enum "clj" "cljs"]]}}
[:map ::name ::title ::age ::skills]]
#{::name ::age})
;[:map
; [:user/name :user/name]
; [:user/title {:optional true} :user/title]
; [:user/age :user/age]
; [:user/skills {:optional true} :user/skills]]#2022-01-0820:17ikitommishould be easy to make it recursive, I think there is an issue to add such a tool to malli.util…#2022-01-0922:59Grant Horner@U055NJ5CC that's definitely simpler than my solution… thanks!#2022-07-1921:16Noah Bogartwhat about this?
(defn select
[schema ks]
(let [schema (m/schema schema)
plain-keys (filter keyword? ks)
map-keys (->> ks
(filter map?)
(apply merge)
(not-empty))
req-keys (vec (concat plain-keys (keys map-keys)))
changed-schema (-> (mu/optional-keys schema)
(mu/required-keys req-keys))]
(if-let [paths map-keys]
(reduce (fn [schema [path v]]
(assert (vector? v) "Nested map vals must be vectors")
(mu/update schema path select v))
changed-schema
paths)
changed-schema)))#2022-07-1921:18Noah Bogartworks like this:
(select
(m/schema
[:map
[:name string?]
[:title [:enum "boss" "not-boss"]]
[:skills [:set [:enum "clj" "cljs"]]]
[:base [:map
[:this-one [:map
[:one uuid?]
[:two int?]]]
[:not-this-one string?]]]])
[:name {:base [{:this-one [:two]}]}])
; [:map
; [:name string?]
; [:title {:optional true} [:enum "boss" "not-boss"]]
; [:skills {:optional true} [:set [:enum "clj" "cljs"]]]
; [:base [:map
; [:this-one [:map
; [:one {:optional true} uuid?]
; [:two int?]]]
; [:not-this-one {:optional true} string?]]]]#2022-01-0820:17ikitommisomething like this?
(require '[malli.core :as m])
(require '[malli.util :as mu])
(defn select [schema keys]
(-> schema
(mu/optional-keys)
(mu/required-keys keys)))
(select
[:schema {:registry {::name :string
::title [:enum "boss" "not-boss"]
::age :int
::skills [:set [:enum "clj" "cljs"]]}}
[:map ::name ::title ::age ::skills]]
#{::name ::age})
;[:map
; [:user/name :user/name]
; [:user/title {:optional true} :user/title]
; [:user/age :user/age]
; [:user/skills {:optional true} :user/skills]]#2022-07-1921:16Noah Bogartwhat about this?
(defn select
[schema ks]
(let [schema (m/schema schema)
plain-keys (filter keyword? ks)
map-keys (->> ks
(filter map?)
(apply merge)
(not-empty))
req-keys (vec (concat plain-keys (keys map-keys)))
changed-schema (-> (mu/optional-keys schema)
(mu/required-keys req-keys))]
(if-let [paths map-keys]
(reduce (fn [schema [path v]]
(assert (vector? v) "Nested map vals must be vectors")
(mu/update schema path select v))
changed-schema
paths)
changed-schema)))#2022-01-0820:43ikitommiGiven a Var:
(defn -instrument
"Takes an instrumentation properties map and a function and returns a wrapped function,
which will validate function arguments and return values based on the function schema
definition. The following properties are used:
| key | description |
| ----------|-------------|
| `:schema` | function schema
| `:scope` | optional set of scope definitions, defaults to `#{:input :output}`
| `:report` | optional side-effecting function of `key data -> any` to report problems, defaults to `m/-fail!`
| `:gen` | optional function of `schema -> schema -> value` to be invoked on the args to get the return value"
([props]
(-instrument props nil nil))
([props f]
(-instrument props f nil))
([{:keys [scope report gen] :or {scope #{:input :output}, report -fail!} :as props} f options]
...
a simple schema inferrer:
(md/infer #'m/-instrument {::md/sequential-maps false})
;[:function
; [:=> [:cat :any] :any]
; [:=> [:cat :any :any] :any]
; [:=>
; [:cat [:map
; [:scope {:optional true} :any]
; [:report {:optional true} :any]
; [:gen {:optional true} :any]] :any :any]
; :any]]#2022-01-0820:43ikitommialso can read the plumatic hints. Could add it to read also normal Clojure typehints.#2022-01-0820:48ikitommiit can parse both the source code and the :arglists meta (both are described as malli sequence schemas). As we can create clj-kondo hints too, now we can do Var -> Malli -> Clj-kondo for free as in 🍺#2022-01-0820:49ikitommiit also understands the new Clojure keyword-ags-as-maps thing and the maps-as-sequence.#2022-01-0820:51ikitommithis is the real schema for it:
(md/infer #'m/-instrument)
;[:function
; [:=> [:cat :any] :any]
; [:=> [:cat :any :any] :any]
; [:=>
; [:cat
; [:altn
; [:map [:map
; [:scope {:optional true} :any]
; [:report {:optional true} :any]
; [:gen {:optional true} :any]]]
; [:args
; [:schema [:* [:alt
; [:cat [:= :scope] :any]
; [:cat [:= :report] :any]
; [:cat [:= :gen] :any]
; [:cat :any :any]]]]]]
; :any
; :any]
; :any]]#2022-01-0910:42ikitommithe plumatic syntax is merged in master, also better dev-tooling for CLJS (thanks to @danvingo!). Looking forward to test reports, will release soon/after.#2022-01-0913:58Noah BogartThat’s so exciting! #2022-01-0917:44metameI’m interested in using malli with clj-kondo for static type checking. We already have clj-kondo in our project, and I saw in the malli readme on how to generate configs. Wondering if there are any decent examples that would demonstrate how to organize this for a large production code base?#2022-01-0917:59Ben SlessStatic type checking in Clojure is a bit limited without type inference, which malli and kondo don't do.
You'll have to specify everything you want checked.
A sane middle ground is having a "schema" namespace where you specify the schema of data coming in or out of your code base, then validate at the edges of the system.#2022-01-0918:06borkdudeclj-kondo does have some type inference#2022-01-0918:08Ben Sless(some type-inference? clj-kondo) ;; => true
How much type inference does it do, exactly?#2022-01-0918:16borkdudeit has return type inference and locals inference for example#2022-01-0918:16borkdude$ clj-kondo --lint - <<< '(let [x :foo] (inc x))'
<stdin>:1:20: error: Expected: number, received: keyword.
#2022-01-0918:17borkdude$ clj-kondo --lint - <<< '(let [x (inc 2)] (assoc x :foo :bar))'
<stdin>:1:25: error: Expected: associative collection or nil, received: number.
#2022-01-0918:20Ben SlessDo associative collections carry around type information about their keys?#2022-01-0918:23borkdudethere is some of this, don't remember exactly how much, but probably not a lot#2022-01-0918:23borkdudethere is one open issue by @U055NJ5CC about this, that I haven't gotten around to yet#2022-01-0919:22metamewe already use plumatic.schema for data validation at the edges. I’m primarily talking about this:
https://github.com/metosin/malli#clj-kondo
I can see how to define the function spec right after with m/=> .#2022-01-0919:23metameWhat I’m wondering is if there’s a best practice for emitting all of the configs for a project after defining the specs in each ns?#2022-01-0919:26metameAnd further than that, would this buy me anything when it comes to associative collections?#2022-01-0919:27metameBut even just using it for arg types and return types, it’s still a step up imo#2022-01-0919:30borkdude@U774Z4VJA you get things for free and it might improve over time if you complain enough in the #clj-kondo channel :)#2022-01-0919:31borkdudeassociative collections: if you specify a required key as input to a function and you pass a map literal, then clj-kondo will be able to see it#2022-01-0919:31borkdudebut the inference on associative collection needs more love#2022-01-0919:31metameha, ya. Maybe if work stay calm enough, I may even be able to contribute here and there 😉#2022-01-0919:32metameAny tips on emitting config for a whole project?#2022-01-0919:33borkdudefor this question I defer to the malli authors!#2022-01-0919:33metameofc thanks#2022-01-0919:34metametl;dr of above thread. One unanswered question:
Any tips on emitting clj-kondo config for a whole project?#2022-01-1014:36ikitommiThere is no guide for that and as there are 3+ ways to declare the schemas for functions, no best practices yet I think. I put the schemas next to the functions (my favourite is the plumatic-syntax, so inlined) and have the ! in the user / dev namespace, so you can do that easily. It instruments all the functions from all loaded namespaces, so should work ok.#2022-01-1014:36ikitommiif not, please report 🙂 planning to mallify a large codebase, can comment after that how it works / doesn’t.#2022-01-1014:37ikitommiif you are doing that already, user experience report post & feedback most welcome#2022-01-1014:39metameMany thanks for the response!
Will ! also emit the clj-kondo config @U055NJ5CC?#2022-01-1014:43ikitommiyes#2022-01-1014:44ikitommihttps://github.com/metosin/malli/blob/master/src/malli/dev.clj#L15-L36#2022-01-1014:44metameThat is 🔥#2022-01-1014:44metameThanks#2022-01-1014:44ikitommi👍#2022-01-1014:48metameI also like the plumatic syntax so very cool that was just added#2022-01-1009:19Jungwoo KimHi I’m thinking of using malli with my project and still investigating it. I have a simple question. What do :cat and :catn means? I want to know the full name of cat and catn to understand better . alt and altn too.#2022-01-1009:40Ben SlessCat is a con*cat*enation of sequences
Alt is an alternation of sequences
The n suffix is "named", like a named capture group in a regular expression#2022-01-1009:41Ben SlessI can elaborate if this is unclear#2022-01-1009:44Jungwoo KimThanks a lot. I understood!#2022-01-1102:07Jungwoo KimHi! I’m really into malli very recently. thanks!
I have a question though.
Here’s my example pants-schema.
(def pants-schema
[:and
[:map
[:id int?]
[:size {:optional true} [:maybe :int]]
[:size-alphabet {:optional true} [:maybe [:enum "S" "M" "L"]]]]
[:fn {:error/message "size and size alphabet should be nil-matched"}
'(fn [{:keys [size size-alphabet]}]
(or (and (nil? size) (nil? size-alphabet))
(and (some? size) (some? size-alphabet))))]])
You can see my nil-matched fn schema. I’m trying to use the schema to validate for functions like below,
(defn do-something-with-pants
{:malli/schema [:=> [:cat pants-schema] :nil]}
[pants]
(prn pants)) ; do something
and then, if i pass the value {:id 1 :size 90} into the do-something-with-pants , I want to extract the error like humanize results but I got the full error messages.
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
; :malli.core/invalid-input {:input [:cat [:and [:map [:id int?] [:size [:maybe :int]] [:size-alphabet [:maybe [:enum "S" "M" "L"]]]] [:fn #:error{:message "size and size alphabet should be nil-matched"} (fn [{:keys [size size-alphabet]}] (or (and (nil? size) (nil? size-alphabet)) (and (some? size) (some? size-alphabet))))]]], :args [{:id 1, :size 90}], :schema [:=> [:cat [:and [:map [:id int?] [:size [:maybe :int]] [:size-alphabet [:maybe [:enum "S" "M" "L"]]]] [:fn #:error{:message "size and size alphabet should be nil-matched"} (fn [{:keys [size size-alphabet]}] (or (and (nil? size) (nil? size-alphabet)) (and (some? size) (some? size-alphabet))))]]] :nil]}
I know explain + humanize is very useful to see the readable results but as I feel like that’s good for development stage. In the web application, My needs are very important to me. How do I handle errors properly?#2022-01-1211:33rovanionAny recommendation on dealing with time? I've got multiple fields that hold LocalDateTime and it's not clear to me how I should express that.#2022-01-1215:24ikitommiyou could c&p code from this https://github.com/metosin/malli/pull/545, or wait for the PR or just do something like:
(def LocalDateTime
(m/-simple-schema
{:type 'LocalDateTime
:pred #(instance? LocalDateTime %)
:type-properties {:error/message "invalid date-time"
:decode/string #(LocalDate/parse %)
:json-schema {:type "string"
:format "date-time"}}))
(m/validate LocalDateTime (LocalDateTime/now)) ; => true
(didn’t run that, might not work, but like that anyways.#2022-01-1816:55rovanionThank you!#2022-01-1212:15rovanionWhen walking a schema, what does the nil that appears as the second entry in all children represent?
(def temperaturmätning
(malli/schema
[:map
[:odlingsplats [string? {:min 1}]]
[:tidpunkt string?]]))
(malli/walk
temperaturmätning
(fn [schema path children _]
(println "Path: " path)
(println "Type: " (malli/type schema))
(print "Schema: ")
(clojure.pprint/pprint schema)
(clojure.pprint/pprint children)
(case (malli/type schema)
string? [:input {:type "text"} (-> path last name)]
:map [:div.input-group children])
))
;; [:div.input-group
;; [[:odlingsplats nil [:input {:type "text"} "odlingsplats"]]
;; [:tidpunkt nil [:input {:type "text"} "tidpunkt"]]]] #2022-01-1212:23ikitommientry properties, e.g. in`[:map [:x {:optional true} :int]]`#2022-01-1213:32rovanionAh, of course!#2022-01-1216:20Ben Sless@ikitommi since the time PR is back on the agenda, what do you think about tackling the protocols split first?#2022-01-1218:06ikitommiThanks for reminding. I’ve been thinking about it. Just commented on the issue.#2022-01-1306:15Mutasem HidmiHi guys, how are you? I am new to Clojure. I wanted to ask a question about humanizing the errors returned from Malli when used as coercion. I tried to do like below, but it didn't work.
(def custom-coercion (reitit.coercion.malli/create
{:transformers {:body {:default reitit.coercion.malli/default-transformer-provider
:formats {"application/json" reitit.coercion.malli/json-transformer-provider}}
:string {:default reitit.coercion.malli/string-transformer-provider}
:response {:default reitit.coercion.malli/default-transformer-provider}}
:error-keys #{:type :coercion :in :schema :value :errors :humanized #_:transformed}
:compile mu/closed-schema
:encode-error (fn [error] {:errors (:humanized error)})
:validate true
:enabled true
:strip-extra-keys true
:default-values true
:options nil}))
#2022-01-1306:27dharriganI have an example#2022-01-1306:28dharrigangive me a moment and I'll push it up to github#2022-01-1306:35Mutasem HidmiThank @dharrigan. I appreciate it.#2022-01-1306:39dharriganHere you go: #2022-01-1306:39dharriganIt's a very simple example, which may help you along the path#2022-01-1306:40dharriganIn particular, look at:#2022-01-1306:40dharrigan#2022-01-1306:40dharriganThen#2022-01-1306:40dharrigan#2022-01-1306:40dharriganAnd here is the malli spec for the API:#2022-01-1306:40dharrigan#2022-01-1306:40dharrigan.#2022-01-1306:42dharriganThe encode-error function can be certainly improved (it assumes only one error, there could be multiple). Treat it like a starting point 😉#2022-01-1306:42dharriganHere is an example of a failure response:#2022-01-1306:42dharrigan❯ http :8080/api/foo
HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
What is going on?? There is a 'missing required key'. I was expecting a value for '[:id]' but got 'null' instead!
#2022-01-1306:42dharriganand here is an example of a successful response:#2022-01-1306:43dharrigan❯ http :8080/api/foo?id=10
HTTP/1.1 200 OK
Content-Length: 14
Content-Type: application/json;charset=utf-8
{
"hello": "10"
}#2022-01-1306:43dharrigan.#2022-01-1306:43Mutasem HidmiThank you. I am checking it now#2022-01-1308:48Mutasem HidmiThanks buddy. By the way, you can put also this :error-keys #{:type :coercion :in :schema :value :errors :humanized #_:transformed} in order to get a humanized key that you can use in the encode-error#2022-01-1416:04metameAnyone have an example of plumatic syntax in the wild (not in docs yet)?
(also possible I’ve misunderstood the meaning of adding plumatic syntax)
I see it in the Destructuring section of the README for fn type hints.
EDIT: looking at the PR, I’m seeing the usage under the experimental ns. So basically we can use in-line plumatic syntax for in-line function types but not anything similar to plumatic syntax for a full schema def. For that we still need to use the vector or map syntax.#2022-01-1416:34ikitommino full schema syntax. That would be... interesting.#2022-01-1416:48metamethanks 🙇#2022-01-1421:05mynomotoIs there someplace where the type options are documented? E.g. [:qualified-keyword {:namespace :xxx}] works but [:qualified-namespace {:namespace :xxx}] doesn´t check the namespace.#2022-01-1820:52coltnzHi, is there a way to default a value even if the key is optional in the schema?#2022-01-1820:52coltnzI have a map which may or may not have a date entry. In the latter case I want to default it to now#2022-01-1820:52coltnz(m/decode [:map
[:date {:optional true}
[:maybe [:string {:decode/string mt/-string->date :default (tick/now)}]]]]
{}
(mt/transformer mt/string-transformer mt/default-value-transformer))#2022-01-1820:53coltnzreturns {} which doesn't surprise me.#2022-01-1823:20Noah Bogart#2022-01-1906:01ikitommi@coltnz no at the moment. But we could add an option to mt/default-value-transformer to fill those, here’s the code for it: https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L394. Issue & PR welcome.#2022-01-1907:06coltnzcool i'll do a PR#2022-01-1916:51rovanionAnyone ever had their data thrown away during coercion in reitit when using their own custom malli schema type?
I've defined mine like:
(defn- -string-gen [{:keys [min max]}]
(cond
(and min (= min max)) (gen/fmap string/join (gen/vector gen/char min))
(and min max) (gen/fmap string/join (gen/vector gen/char min max))
min (gen/fmap string/join (gen/vector gen/char min (* 2 min)))
max (gen/fmap string/join (gen/vector gen/char 0 max))
:else gen/string))
(def postgres-string
"Postgres does not allow the null byte \0 in strings.
The predicate only forbids null bytes, but the generator only generates ascii characters. This
severely limits the range of test strings. The latter should be changed at some later point."
(malli/-simple-schema
(fn [opts _]
{:type :postgres/string
:pred #(and (string? %) (not (string/includes? \0 %)))
:type-properties {:error/message "Unable to decode postgres string"
:decode/string (fn [x] (prn "We are in the decoder") x)
:json-schema/type "string"
:json-schema/format "string"
:json-schema/minimum 0
:gen/gen (-string-gen opts)}})))
But when defining a schema with it like
(def kasse
(malli/schema
[:map
[:odlingsplats [postgres-string {:foreign-key "odlingsplatser"}]]
[:diameter_m int?]
[:djup_m int?]]))
and then running a request through my router like
(tove.handler/app-routes {:request-method :post, :uri "/kassar/-/new" :form-params {:odlingsplats "En bra plats" :diameter_m "1" :djup_m "1"}})
the data is without any error conformed into
{:diameter_m 1, :djup_m 1}
If I replace postgres-string with string? in the schema the data gets passed on and the conformed map looks like:
{:odlingsplats "En bra plats", :diameter_m 1, :djup_m 1}#2022-01-1916:54Ben SlessHow did you define coercion for the router?#2022-01-1916:57rovanionHere's the relevant section of the router:
["/-/new" {:post {:handler (partial page/create-entry! "kassar" "id")
:coercion mc/coercion
:parameters {:form s/kasse-user-data}}}]]
I also have this at the very end of the router definition:
{:data {:middleware [rrc/coerce-request-middleware]}}#2022-01-1917:03Ben SlessI think you need to specify it should use string coercion#2022-01-1917:32rovanionI'm not sure I understand, where should I tell what what?#2022-01-1917:33Ben SlessI'll dig up an example#2022-01-1917:34Ben SlessI think you need something like
(mcoercion/create
{:transformers
{:body
{:default string-transformer-provider
:formats {"application/json" string-transformer-provider}}
:string
{:default string-transformer-provider}
:response
{:default mcoercion/default-transformer-provider}}})
#2022-01-1917:34Ben SlessRoughly#2022-01-1917:35Ben Slessidea - use string coercion instead of json coercion by default for your params#2022-01-2009:22rovanionWhat seems to have worked is that I set up my own spec-style mutable registry that I set as the default registry. And by referring to the type as a key in kasse instead of as a var it all worked out.#2022-01-2009:23rovanionNow my "We are in the decoder" message is printed and all is fine!#2022-01-1919:55Noah BogartIs there a shorter form of
(def FxSchema
(m/schema
[:* [:or
nil?
[:tuple qualified-keyword?]
[:tuple qualified-keyword? [:maybe any?]]]]))
#2022-01-1920:35Ben Slessor nil? T -> maybe T#2022-01-1920:36Ben Sless[:sequence qualified-keyword? [:? any?]]#2022-01-1920:37Ben Sless[:* [:maybe [:sequence qualified-keyword? [:? any?]]]]#2022-01-1920:37Noah Bogartvery nice, thank you#2022-01-1920:40Noah Bogarthm, that doesn’t work because :sequential requires homogenous input. maybe i need :cat?#2022-01-1920:41Noah Bogartthat’s it: :sequential -> :cat works#2022-01-1920:41Noah Bogartthanks for the help!#2022-01-1920:45Ben Sless👍#2022-01-1920:45Ben SlessWhen in doubt, grab a catjam#2022-01-2004:57Abhinavhow do I express this in a schema?
{"hi" {:greeting "hi" :id 1} "hello" {:greeting "hello" :id 2} "hiee" {:greeting "hiee" :id 3}}
I have a function that transforms a map like this
[{:greeting "hi" :id 1}{:greeting "hello" :id 2}{:greeting "hiee" :id 3}]
into the one above, the values of :greeting aren’t known before hand#2022-01-2011:16ikitommimaybe: [:map-of :string [:map [:id :int] [:greeting :string]]]#2022-01-2011:23AbhinavThat is a good suggestion, but I want to say that the key of the map has to the be equal to the value of the :greeting key.
so [:map-of :string [:map [:id :int] [:greeting :string]]]
would be valid for
{"foostring" {:greeting "hi" :id 3}}
but the above won’t be valid#2022-01-2011:24AbhinavI don’t know if what I’m tryign to do is an anti-pattern:sweat_smile:#2022-01-2011:37ikitommino, it’s all valid, maybe:
(def Schema
(m/schema
[:and
[:map-of :string [:map [:id :int] [:greeting :string]]]
[:fn (partial every? (fn [[k d]] (= k (:greeting d))))]]))
(m/validate
Schema
{"hi" {:greeting "hi" :id 1}
"hello" {:greeting "hello" :id 2}
"hiee" {:greeting "hiee" :id 3}})
; => true
(m/validate
Schema
{"hi" {:greeting "<<invalid>>" :id 1}
"hello" {:greeting "hello" :id 2}
"hiee" {:greeting "hiee" :id 3}})
; => false#2022-01-2011:39ikitommim/explain here doesn’t produce correct place of error, could should do it, we could tune either :fn to support returning custom explain-results, or add support for generic :explain key for keys, or something. Someone should write an issue out of this.#2022-01-2103:25Abhinav@U055NJ5CC thank you for your help. that was exactly what I was looking for.#2022-01-2012:37rovanionI believe I'm just being daft here but I haven't been able to figure this out for more than an hour so: How do I resolve a keyword to a schema in the registry?
(def registry
(atom {}))
(defn register! [type ?schema]
(swap! registry assoc type ?schema))
;; Combine the default registry with our own mutable registry.
(mreg/set-default-registry!
(mreg/composite-registry
(mreg/fast-registry (malli/default-schemas))
(mreg/mutable-registry registry)))
(register! :db/kasse
[:map
[:id [:int {:primary-key true :db-generated true}]]
[:odlingsplats [:string {:foreign-key "odlingsplatser"}]]
[:diameter_m :int]
[:djup_m :int]
[:volym_m2 [:int {:db-generated true}]]])
(malli/walk
:db/kasse
(malli/schema-walker identity))
;; => :db/kasse
I've tried wrapping :db/kasse in different functions from malli but none seem to do the lookup. The lookup function in the core namespace is private. Just running (:db/kasse malli/default-registry) does not work either. (malli/schema :db/kasse) seems like the obvious choice but it seemingly has no effect.
(malli/walk
(malli/schema :db/kasse)
(malli/schema-walker identity))
;; => :db/kasse
#2022-01-2012:58ikitommithe :db/kasse returned is a Malli Schema instance, it’s print output is just the form, so looks like keyword. It’s type is :malli.core/schema, which is the internal eager reference, like a Var in Clojure. If you want to get the schema behind it, you can m/deref it. But, calling m/validate on :db/kasse works too. the :malli.core/schema forwards the calls to the actual instance, like Var. Hope this helps.#2022-01-2012:59ikitommialso, walking it just work, try something like (m/ast :db/kasse) to verify#2022-01-2013:58rovanionThank you, dereffing worked perfectly. Though I'm not sure I understand the latter reply:
(let [schema (malli/schema :db/kasse)]
(prn (type schema))
(malli/walk
schema
(malli/schema-walker identity)))
;; => :db/kasse
;; printed: :malli.core/schema
#2022-01-2016:49mafcocincohi all! Just started using malli and I have what I think is a pretty newbie question: Is there a way, in the context of a :map schema declaration to have mutually exclusive keys? That is “map should contain :a and :b together or :c by itself”.#2022-01-2017:39Ben SlessNot atm, but it's an often requested feature#2022-01-2018:21jeroenvandijkHi, I’ve been playing with the idea of using Malli as a way to define the main data model of an application. Many features are already in Malli I believe. One thing that might be missing is an easy way to generate consistent test and seed data. Maybe something like composite types are an useful addition?
(register! :user/first-name [:enum "john" "joe" "alex"])
(register! :user/last-name [:enum "smith" "wood" "lee"])
(register! :user/full-name
[:composite {:schema string?
:compose (fn [first-name last-name]
(str first-name " " last-name))
:fields [:user/first-name :user/last-name]}])
(register-entity! ::user
[:user/first-name
:user/last-name
:user/full-name])
(generate ::user)
;; => #:user{:first-name "joe", :last-name "wood", :full-name "joe wood"}
More here https://gist.github.com/jeroenvandijk/5e0785f25f7fdfeac7bc7a0be72cb62a#file-malli_composite_test-clj#2022-01-2019:39Ben SlessDid you just implement unification?#2022-01-2019:56jeroenvandijkNot consciously 😅 Maybe accidentally? I was just thinking how do I get meaningful seed and test data when I’m just defining my attributes and entities#2022-01-2020:09jeroenvandijk@UK0810AQ2 I see you did some work on unification yourself (https://github.com/metosin/malli/issues/474 and https://github.com/bsless/malli-keys-relations). My approach is not that advanced (or complete)#2022-01-2020:14Ben SlessIt's okay, my approach sucked anyway
I'm becoming convinced the only solution is embedding micro kanren and I'm scared#2022-01-2020:21jeroenvandijkHaha I can imagine#2022-01-2110:46Karol WójcikDoes malli supports inline schemas in map destructuring syntax in defn? e.g.,
(mx/defn kikka
[{:keys [
hello :- :string
world :- :string
]}]
(println hello world))#2022-01-2110:48Karol WójcikIt looks like not.
:schema
[:=>
[:cat
[:altn
[:map
[:map
[:hello {:optional true} :any]
[:- {:optional true} :any]
[:string {:optional true} :any]
[:world {:optional true} :any]]]
[:args
[:schema
[:*
[:alt
[:cat [:= :hello] :any]
[:cat [:= :-] :any]
[:cat [:= :string] :any]
[:cat [:= :world] :any]
[:cat :any :any]]]]]]]
:any],#2022-01-2113:06ikitommididn’t add that as the current support is “plain plumatic”.#2022-01-2113:06ikitommibut, plumatic doesn’t support different returns from arities, so we should extend it anyway#2022-01-2113:07ikitommibut, if that is supported, one can’t have a key named -, right?#2022-01-2113:14ikitommi(let [{:keys [a :- :int]} {:a 1, :- 2, :int 3}] [a - int])
; => [1 2 3]#2022-01-2113:16ikitommiso, don’t think it’s a good idea to support it, what do you think?#2022-01-2116:43Karol WójcikRight. After consideration I also think the syntax where the schema is specified in keys destructuring, is pretty hard to understand. #2022-01-2202:49jkrasnayIs there a way to get function instrumentation working in CLJS? There seems to be some stuff in malli.instrument.cljs but my CLJS build complains that that ns is not available.#2022-01-2207:30ikitommiHaven’t used that myself, but the tests pass. Are you depending on the latest version from master? It’s not released yet#2022-01-2215:22jkrasnayOh, I was on 0.7.5. Thanks!#2022-01-2220:19dvingo@U0DTSCAUU feel free to ping me if you run into any issues!#2022-01-2207:30ikitommiadded .pretty/explain#2022-01-2211:22ikitommifrom cursive#2022-01-2211:21ikitommipushed all current stuff for leiningen users as`[metosin/malli "0.8.0-SNAPSHOT"]`#2022-01-2213:36Karol Wójcik@ikitommi FYI: The CLJ Kondo config has unbalanced bracket 🧵#2022-01-2213:36Karol WójcikA PR with issue fixed https://github.com/metosin/malli/pull/625#2022-01-2214:32ikitommiThanks! Merged#2022-01-2215:28jkrasnayI’m having a problem with instrumenting my functions. I define a function in namespace A and register it with Malli as follows:
(m/=> selectable-label [:=> [:cat [:map
[:build map?]
[:selectable-id id/SelectableId]]]
locale/LocalizableString])
Then in namespace B I call
(mi/instrument!)
This triggers errors when compiling namespace B that namespaces id and locale are not found. These are aliases from the :require block in namespace A. Is there any way around this?#2022-01-2215:42jkrasnayBTW namespace A is cljc while namespace B is cljs.#2022-01-2215:46jkrasnayIt works if I fully qualify id and locale.#2022-01-2216:26jkrasnayI think this may be one of those subtle things about CLJS macros, where we can’t qualify the namespace since the macro is running in CLJ.#2022-01-2216:58Игорь ЛисовцовHi there guys. I'm new to clojure and especially to malli.
So, I'm playing around repl and malli, and notices that below example from the documentation doesn't work
(m/decode
[string? {:decode {:string 'str/upper-case}}]
"kerran" mt/string-transformer)
; => "KERRAN"
and instead of "KERRAN" it returns "kerran". Func at the place {:string func simply not called.
Did I do something wrong?#2022-01-2217:12ikitommiThe docs for this part are ahead of the release, plan is to release 0.8.0 n the next few days. If you know how to depend on the latest commit of malli, you can test that too.#2022-01-2217:18Игорь ЛисовцовGot it. Thank you a lot!#2022-01-2308:56pithylessI feel I'm missing something obvious; if I've got a [:map [:name :string] [:age :int]] schema, what's the best way to get back the coll of keys [:name :age]?#2022-01-2309:26pithylessLooking at the transformers, I guess it would be:
(some->> schema m/entries (map first) seq)
#2022-01-2312:05ikitommiyes, that's it.#2022-01-2316:03ikitommi🥳#2022-01-2422:23lsenjovThis is the real Christmas present I’ve been waiting on 🥳#2022-01-3001:39Grant HornerThis is awesome! I never used schema, and I'm loving the new malli.experimental/defn. How do you make it play nicely with Cursive? I've been unable configure the cursive resolution to get rid of all the yellow squigglies#2022-01-3007:22ikitommithanks! You should mark it to be resolved as schrma.core/defn. And go vote up https://github.com/cursive-ide/cursive/issues/2645#2022-01-2719:55Noah BogartI have a map that contains function values, but I want to use a Var (`#'function-name`) to facilitate repl-driven development, however fn? doesn’t consider the Var a function and malli doesn’t recognize var? as a schema function#2022-01-2720:14Ben SlessYou want to validate the map itself?#2022-01-2720:15Noah BogartI have the validation for the map, but i’m realizing I need to be able to say [:or fn? var?] instead of just fn?#2022-01-2720:15Ben SlessYou can use a decoder#2022-01-2720:16Ben SlessAnd deref the values :)#2022-01-2720:17Ben SlessI.e. [:map [:k [fn? {:decode/var deref}]]]#2022-01-2720:17Ben SlessApproximately#2022-01-2720:17Noah Bogarthuh interesting. i’ll give that a shot! thanks#2022-01-2720:19Ben SlessCheers. Make sure to wrap that deref in a try catch#2022-01-2720:19Ben SlessBut, why are you validating a map of functions? Especially w/o function schema it isn't very informative, is it?#2022-01-3118:07Noah Bogartjust barebones checking that i’m creating the right types, not doing much more than that to start#2022-01-2719:56Noah Bogartany way to handle this?#2022-01-3117:55Brett Rowberrywhen inferring malli schemas from data, I can tell it how to handle strings as uuids, for example, as shown here
https://github.com/metosin/malli/pull/597
some string values may be inst or other types - is there a way to have it handle those as well?#2022-01-3118:15ikitommisure, anything you can present transformers works here. Check how inst? is handled in malli.transform#2022-01-3119:16ikitommihttps://github.com/metosin/malli/pull/632#2022-02-0117:53Brett Rowberrysorry, I should have given a better example
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])
(mp/provide
[{:my-uuid "caa71a26-5fe1-11ec-bf63-0242ac130002"
:my-inst "2021-01-01T00:00:00Z"}]
{::mp/value-decoders {'string? {:uuid mt/-string->uuid}}})
;; can I treat :my-uuid as a uuid and :my-inst as an inst?#2022-02-0118:55ikitommiwith master:
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])
(mp/provide
[{:my-uuid "caa71a26-5fe1-11ec-bf63-0242ac130002"
:my-inst "2021-01-01T00:00:00Z"}]
{::mp/value-decoders {'string? {:uuid mt/-string->uuid
'inst? mt/-string->date}}})
; => [:map [:my-uuid :uuid] [:my-inst inst?]]#2022-02-0121:46Brett Rowberrythank you!#2022-02-0123:00Brett Rowberryso this would be after 0.8.0#2022-02-0207:41ikitommiif you use deps, you can depend on the lastest commit on master to try this out. for leiningen - in the next release.#2022-02-0214:11Brett Rowberrywe're using some custom bazel thing, so I'll have to wait for the jar. thank you so much for this feature! type providers were so awesome in F# - I really appreciate having something similar in Clojure!#2022-02-0513:01ikitommi[metosin/malli "0.8.1"]#2022-02-0513:01ikitommiand good to hear, it is useful 🙂#2022-02-0118:55ikitommiwith master:
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])
(mp/provide
[{:my-uuid "caa71a26-5fe1-11ec-bf63-0242ac130002"
:my-inst "2021-01-01T00:00:00Z"}]
{::mp/value-decoders {'string? {:uuid mt/-string->uuid
'inst? mt/-string->date}}})
; => [:map [:my-uuid :uuid] [:my-inst inst?]]#2022-02-0117:50Grant HornerIs there a way to truncate the output of the pretty reporter? I’m getting occasional reports for functions that take in vectors of 10,000+ elements and the output is absolutely crushing my poor intellij repl#2022-02-0118:53ikitommilooking at fipp docs, try setting clojure.core/*print-length* & clojure.core/*print-level*, https://github.com/brandonbloom/fipp#printer-usage.#2022-02-0118:53ikitommiif that works, doc PR to malli welcome#2022-02-0217:23Grant HornerJust did! I appreciate the quick response, I could’ve figured it out if I had read the documentation more in-depthly :face_palm:#2022-02-0217:27ikitommiThanks! Merged#2022-02-0314:32lreadJust noticed this in my vid feed: https://youtu.be/mNpE7cPm-N8#2022-02-0408:57JacquesIs it possible to use registry references for function schema input parameters? Is there an example somewhere to see how this might work?#2022-02-0513:16ikitommi@jacquesdpz maybe:
(require '[malli.core :as m])
(require '[malli.registry :as mr])
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
{::user-id :int
::user [:map
::user-id
[:name :string]
[:age :int]]}))
(require '[malli.experimental :as mx])
(require '[malli.generator :as mg])
(mx/defn get-user :- ::user [id :- ::user-id]
(assoc (mg/generate ::user) ::user-id id))
(get-user 123)
; => {:kikka/user-id 123, :name "8p28ySz", :age 2405534}#2022-02-0709:13Jacquesthank you @U055NJ5CC#2022-02-0604:23pinkfrogIs it possible to specify the :min/:max attributes for regular expressions (instead of doing it inside the re pattern)?#2022-02-0608:00ikitommino. Regal (https://github.com/lambdaisland/regal) has malli-support, but not sure it's up-to-date with the latest version.#2022-02-0714:44Ben SlessWhere does it make sense in your opinion to specify keywords renaming for a transformer? in a map schema's property or on its entry?#2022-02-0714:45Ben Sless[:map {:rename {:a :b}} [:b int?]] vs [:map [:b {:rename :a} int?]]#2022-02-0716:40ikitommiMy intuition says it belongs to the map. But I guess, it depends. Being part of the entry-tuple, you could for example merge two maps with mapping to same domain and the mappings would be merged automatically#2022-02-0716:41ikitommie.g.
[:merge
[:map [:a {:rename/SAP "SAP_A", :rename/SALESFORCE "sf_a"} int?]]
[:map [:b {:rename/SAP "SAP_B"} int?]]]#2022-02-0716:42ikitomminot actually sure if the entry properties get merged correctly here, just guessing 🙂#2022-02-0717:54Ben SlessFollowup harder question - if I rename in decode it reports error on the wrong key path!#2022-02-0717:54Ben SlessI dug myself a deeper hole than expected 😄#2022-02-0717:54Ben SlessHow do I get out?#2022-02-0806:27Ben SlessThe only way I see around it is attaching as metadata the rename mappings and encode the error report on response#2022-02-0818:01Ben SlessBump / help?#2022-02-0819:00ikitommicould you gist a minimal case, I can try to figure out howto#2022-02-0819:05Ben SlessSure, I'll send it over later today / tomorrow morning#2022-02-0909:05Ben Sless(defn key-renamer
"Take a tuple of keys [`k1` `k2`] and return a function of a map `m`
which will replace `k1` with `k2` if it exists in `m`"
[[k1 k2]]
(fn -rename [m]
(if-let [[_ v] (find m k1)]
(dissoc (assoc m k2 v) k1)
m)))
(defn- -compile-rename-keys-transformer
"Takes a map of keys->keys and returns a function which will rename the
keys in LHS to RHS if they exist in a map."
[m]
(if (seq m)
(reduce comp (map key-renamer m))
identity))
(def rename-keys-transformer
(mt/transformer
{:decoders
{:map
{:compile
(fn [schema _]
(-> schema m/properties :rename -compile-rename-keys-transformer))}}}))
(comment
(m/decode
[:map
{:rename {:a :b :c :d :x :y}}
[:b int?]
[:c int?]
[:y int?]]
{:a 1
:c 2
:x 3}
(mt/transformer rename-keys-transformer)))
Given that an input is incorrect post transformation, how do I report the original field name as invalid?#2022-02-0909:05Ben Sless@U055NJ5CC#2022-02-1110:05ikitommihad time to look at this. Thing is, you should describe the resulting schema, not the orginal.#2022-02-1110:06ikitommiso, it should be:
[:map
{:rename {:a :b, :c :d, :x :y}}
[:b int?]
[:d int?]
[:y int?]]#2022-02-1110:07Ben SlessThe problem is that schemas, being a description of "ought", can't report back errors about "is" for any lossy transformation#2022-02-1110:08Ben SlessI had a typo in the last entry which I can spot now#2022-02-1110:10Ben SlessAnyway, schema validates b, user gave me a, I want to report back "a should be an integer" if I was given a, even thought I treat it as b.
Some sort of alias mechanism#2022-02-1110:10ikitommiok, that :thinking_face:#2022-02-1110:11ikitommiwhat if the rename-map is used to create the target (or source) schema?#2022-02-1110:11ikitommie.g. transform keys.#2022-02-1110:11ikitommithen, you could validate them separately.#2022-02-1110:18ikitommi(def Target
(m/schema
[:map
[:b int?]
[:d int?]
[:y int?]]))
(def mappings {:b :a, :d :c, :y :x})
(def Source
(mu/transform-entries
Target
(partial map (fn [[k p v]] [(mappings k k) p v]))))
Source
; => [:map [:a int?] [:c int?] [:x int?]]#2022-02-1110:19ikitommisomething like that, explicit schemas for both Source and Target.#2022-02-1110:20ikitommiI would like to see someone cook Meander and Malli together, so that one could write a Malli schema and a Meander transformation for it and infer the target Malli schema from those.#2022-02-1110:20ikitommimight be the ultimate data transformation tool 😎#2022-02-1110:21Ben SlessI started off with a separate schema, I wanted to be clever and unify#2022-02-1110:22Ben SlessRegarding meander, I also thought on using it for the complex unification problems I made for myself#2022-02-1110:23Ben SlessAdd a "match" property to schema which binds, then compile that to meander pattern and match on it#2022-02-1110:24Ben SlessOr going the other way, write a meander to malli compiler#2022-02-1110:24Ben SlessThen schemas reflect how data looks#2022-02-1110:25Ben SlessAnd you can go crazy with unification#2022-02-0716:09AJ JaroWhat is the best way to setup the schema to work with a protocol? We could potentially use the https://github.com/metosin/malli#fn-schemas to work through instance?, but maybe there’s a better solution#2022-02-0716:31ikitommiSee https://github.com/metosin/malli/issues/555#2022-02-0815:00AJ JaroThanks. We’ll probably implement some fn schema to check for satisfies or instance for now I guess!#2022-02-0720:20Ben Wadsworth#2022-02-0720:27Ben Wadsworthdoh’ i mean the obvious here is wrong…one sec..#2022-02-0720:28Ben Wadsworth(being the map in one call and string in another) lol…#2022-02-0720:35Ben WadsworthI compounded a couple of issues but I dont think its Malli…. I think it might be coercion from reitit. Sorry bout that, another pair of eyes found the painfully obvious string where a map should have been in my example 😐#2022-02-0721:04Ben Wadsworthyeah no issue… happy monday#2022-02-0818:01Ben SlessBump / help?#2022-02-0811:10Mutasem HidmiHow can set field to be not required in malli Coercion#2022-02-0813:28dvingodoes {:optional true} not work?#2022-02-0814:39pinkfrogI am seeing this coercion failure, but dunno why it fails. It claims missing key, however the key is there.#2022-02-0814:39pinkfrog{
"schema": "[:map {:closed true} [:email [:re #\"^#2022-02-0815:19dvingovalue has email as string not a keyword#2022-02-0815:37pinkfrogYes.#2022-02-0817:58CaseyIs it possible to instrument multimethods with the existing instrumentation features in malli?#2022-02-0818:00Ben SlessWon't multi schema with the dispatch fn itself cover it?#2022-02-0818:24CaseyPerhaps yes.. but then I have to define all schemas on multi-schema on the dispatch fn itself. I'd rather keep the schemas with the defmethod rather than defmulti.#2022-02-0818:58ikitommiHow would you like to define the schemas with defmethod?#2022-02-0822:04CaseyThat's a good question.. defmethod doesn't support metadata afair..only the defmulti does. :thinking_face: Hmm#2022-02-0821:55crimeministerI've got a user-supplied value stored in a map that is a malli schema. I would like to validate that the schema itself is valid; is there a predefined way to do so, or should I author my own "meta-schema" for checking my schema data?#2022-02-0917:31bringeHello. I’m wondering how I can update a schema to dissoc a prop if the schema uses :and at the top-level. For example, if I have the following:
(def my-schema
[:and
[:map
[:id uuid?]
[:some-prop int?]
[:some-prop-2 int?]]
[:fn {:error/message "Some prop must be less than some prop 2"}
(fn [{:keys [some-prop some-prop-2]}]
(< some-prop some-prop-2))]])
and I want to dissoc :id from the map, but keep everything else in the schema the same, how can I do that? I see the malli.util dissoc function and others, but I’m not sure how to use them when :and is involved.#2022-02-0917:46bringeAh, it seems that (mu/update my-schema 0 mu/dissoc :id) works, but maybe there is a better way?#2022-02-0918:08crimeministerThere's a function for manipulating properties that might be worth looking at in case it makes intent clearer: (mu/update-properties).#2022-02-0919:23pithylessfor mu/merge there is a kind of special case:
> * for :and schemas, the first child is used in merge, rest kept as-is
I wonder if that could (and should) be generalized to other malli.util operations#2022-02-1003:22bringe@U066SF64X I’m not sure if update-properties can be used directly in this case, given the :and , but maybe there’s a good way to use it in combination with another operation to make intent clearer.
@U05476190 Yeah :thinking_face:.#2022-02-0920:08Oliver MarksHI, I have this spec which is being used by a reitit route, I am having issues where it fails because the value is a string instead of a keyword, I have been trying to use the json transformer but I think it fails because its not specifically a keyword test, any suggestion on how I could solve this ?
[:or
[:map {:closed true}]
[:map {:closed true}
[:start [:enum :january :feburary :march :april :may :june :july :august :september :october :november :december]]
[:end [:enum :january :feburary :march :april :may :june :july :august :september :october :november :december]]]]
#2022-02-0920:22Ben SlessYou need to add a decoder#2022-02-0920:28Oliver MarksSo i am running the spec through the json-transformer decoder but I dont think it handles this case because its of enum type not keyword#2022-02-0920:29Oliver MarksI am currently looking at -json-decoders trying to see if I can add to that some how#2022-02-0920:32Oliver Marks[:category/type {:optional true} keyword?]
currently the above is working as intended, feels like i need to extend some where I tried adding [:and keyword? [:enum :one :two]] but that does not trigger the conversion#2022-02-0921:19Grant HornerIs there a :seqable schema that would allow you to specify the children’s type? I’m running into an issue where I’d like to have a function accept a vector, list or set of a particular entity, but haven’t been able to figure out how to build the correct collection schema for it. I’m assuming I should be using malli.core/-collection-schema to add a custom schema to the registry#2022-02-1108:12ikitommiYou can also just define the Schema as Var as use it without registering, like Reagent compoents:
(def Seqable (m/-collection-schema {:type 'Seqable, :pred seqable?}))
(m/validate [Seqable :int] '(1 2 3)) ; => true#2022-02-1108:12ikitommiIf you feel that is important, please write an issue to add that to malli.#2022-02-1108:14ikitommiadding new types means the following:
• add it to the core
• generator for it
• JSON Schema (and optionally swagger) mappings
• default humanized error message
• clj-kondo mappings
• + tests#2022-02-1108:16ikitommiwithout registering it, you won’t can’t deserialize a serialized schema, but for anything else, it should work normally.#2022-02-1005:42ZaymonHello. Is there a way to make the exception-info created by Malli coercion less verbose? Currently 500 lines of output to the terminal is a bit hard to parse, mostly because the exception info includes a lot of information about the request.
Either that or a better way to print only the relevant parts of the exception would also be appreciated.#2022-02-1005:46Zaymon#2022-02-1005:58Zaymon(def malli-coercer
(malli-coercion/create
{;; set of keys to include in error messages
:error-keys #{:humanized
;; :type
;; :coercion
;; :in
:schema
:value
:errors
#_:transformed}
;; schema identity function (default: close all map schemas)
:compile mu/closed-schema
;; validate request & response
:validate true
;; top-level short-circuit to disable request & response coercion
:enabled true
;; malli options
:options nil
:muuntaja formats/instance}))
My Coercion is setup like this#2022-02-1019:33emccuemalli/humanize?#2022-02-1101:43ZaymonNot sure how to apply that to exception-info#2022-02-1712:07jussiPondered the same the other day, ended up with simplistic helpers
(defn- path->str
[path]
(string/join " -> " (map #(if (number? %)
(str "[" % "]")
(name %))
path)))
(defn pretty-schema-error
"Make schema errors a bit more readable. Handles nil's and formats path."
[error]
(let [path (:path error)
type (:type error)]
{:path (path->str path)
:type (if (nil? type) "nil" (name type))}))
Using it in our reitit API when handling :reitit.coercion/request-coercion errors for example.#2022-02-1108:17ikitommiwould this make sense? https://clojureverse.org/t/using-schema-like-schemas-in-malli/8613/2?u=ikitommi#2022-02-1108:37jeroenvandijkLooks useful!
- a small step for people acquinted with Plumatic Schema to try Malli
- offers potentially a soft migration path from Schema to Malli. Assuming enough/all Schema predicates will be covered
- extend existing Schema code bases with the power of Malli (I believe :multi will have better error reporting than schema.core/either for instance). This again assumes that the translation of Schema will be near to perfect#2022-02-1108:18ikitommi(require '[malli.experimental.lite :as l])
(def Schema
(l/schema
{:map1 {:x int?
:y [:maybe string?]
:z (l/maybe keyword?)}
:map2 {:min-max [:int {:min 0 :max 10}]
:tuples (l/vector (l/tuple int? string?))
:optional (l/optional (l/maybe :boolean))
:set-of-maps (l/set {:e int?
:f string?})
:map-of-int (l/map-of int? {:s string?})}}))
;[:map
; [:map1
; [:map
; [:x int?]
; [:y [:maybe string?]]
; [:z [:maybe keyword?]]]]
; [:map2
; [:map
; [:min-max [:int {:min 0, :max 10}]]
; [:tuples [:vector [:tuple int? string?]]]
; [:optional {:optional true} [:maybe :boolean]]#2022-02-1108:19ikitommi18 lines of optional sugar, for simple cases like defining route parameters with reitit.#2022-02-1108:46jeroenvandijkI can imagine this dsl would not cover complex cases of Malli, but sure looks useful to remove some boilerplate. Maybe it becomes confusing if these way of schema writing get mixed, not sure#2022-02-1108:55ikitommiyes, I would. not use this for anything “normal”, but for specific / inline cases, good:
default reitit+malli:
:parameters {:query [:map
[:x int?]
[:y {:optional true} string?]]}
adding reitit+malli support for lite too:
:parameters {:query {:x int?, :y (l/optional string?)}}#2022-02-1109:15jeroenvandijkI wanted to say maybe instead of supporting lite directly, asking the user to use to dsl themselves might lead to less confusion. But I see malli can go into malli-lite and malli-lite can go into malli :)
:parameters {:query {:x int?, :y (l/optional [:map-of string? string?]]}
Generates a valid malli schema, and also the other way around:
:parameters {:query [:map
[:x int?]
[:y {:optional true} (map-of string? string?]]}
Nice 💪#2022-02-1109:18jeroenvandijkSo that actually means complex cases are also covered, because normal malli can be used where necessary?#2022-02-1109:59ikitommiyes.#2022-02-1114:15ikitommimalli.experimental.lite now merged in master.#2022-02-1109:59Ben Slesshttps://www.asyncapi.com/ 👀#2022-02-1317:42ikitomminot sure how this relates to malli, but 👍#2022-02-1317:52Ben Slessmostly, thinking about a good way to bring this together with malli, hopefully finding a lossless translation between the two. AsyncAPI looks like a very interesting way to solve the "schema problem" at an organizational level + all the metadata I would have had to reinvent the wheel to specify, which could go beyond interfacing with malli, but further code generation. You could derive reitit routes from it, and more#2022-02-1114:15ikitommimalli.experimental.lite now merged in master.#2022-02-1115:54Brett RowberryToday, when humanizing a value against a map schema, the error message "invalid type" isn't super useful. I know I can add a custom error message. How could it be more descriptive?
(malli.error/humanize (malli.core/explain [:map] "string"))
;; => ["invalid type"]
#2022-02-1116:18dvingohttps://github.com/metosin/malli#custom-error-messages#2022-02-1122:22Brett RowberryRight, I know I can do that for a given schema, but wondered if maybe all maps could say something more specific.#2022-02-1211:41ikitommiYou can override the default error for invalid type, using options to me/humanize.#2022-02-1211:42ikitommithere should be an example behind the link..#2022-02-1212:34dvingooh woops sorry @U021RHDFFHN overlooked that you said that.
I learned something, just tried this out and it works:
(malli.error/humanize
(malli.core/explain [:map] "string")
{:errors (-> default-errors
(assoc ::m/invalid-type
{:error/fn (fn [{:keys [value schema] :as in} _]
(str "The value you provided: '" value "' is not the correct type for the schema: '" (m/form schema) "'"))}))})
=> ["The value you provided: \"string\" is not the correct type for the schema: ':map'"]
#2022-02-1317:41ikitommithe (m/form schema) could be (m/type schema), just report on type, not whole form. otherwise, 💯#2022-02-1418:16Brett RowberryThanks! This works best for my use case:
(malli.error/humanize (malli.core/explain [:map] nil))
;; => ["invalid type"]
(malli.error/humanize
(malli.core/explain [:map [:hi string?]] nil)
{:errors (-> malli.error/default-errors
(assoc ::m/invalid-type
{:error/fn (fn [{:keys [_value schema]} _]
(str "The value provided does not conform to schema: '" (m/form #_m/type schema) "'"))}))})
;; => ["The value provided does not conform to schema: '[:map [:hi string?]]'"]#2022-02-1511:00Martynas MHow does one install sci as an external dependency?
I try to run tests of malli and they fail with this error: :malli.core/sci-not-available
Should I include it into my ~/.clojure/deps.edn?#2022-02-1511:00Martynas MThis was the info from #babashka channel. But it's not enough context.#2022-02-1511:01borkdudehttps://github.com/metosin/malli#running-tests#2022-02-1511:01borkdudeIt seems you need the :test alias#2022-02-1511:02borkdudeI recommend upgrading SCI to 0.3.0 for better performance#2022-02-1511:03Martynas MHow would one run tests from emacs(spacemacs+CIDER)? If I understand correctly you'd need to run it a JS-Clojure REPL. It failed for me when I used Java-Clojure REPL.#2022-02-1511:07borkdude@invertisment_clojuria You can place a file .dir-locals in the repo:
((clojure-mode
(cider-clojure-cli-aliases . "test")))
#2022-02-1511:07borkdudeand then cider-jack-in , after that you should be able to run tests from CIDER#2022-02-1511:08borkdudeif you do not want to add the file, you can do it like this:
C-u M-x cider-jack-in
#2022-02-1511:08borkdudeand then choose clojure-cli and then add the test alias here: -M:test:cider#2022-02-1511:08borkdudeand press enter#2022-02-1511:08Martynas Mok.#2022-02-1511:11Martynas MWhat should I choose when it says "connect sibling cljs"? There are many options. node, figwheel, shadow and others#2022-02-1511:12borkdudeDunno, I don't ever use this. If I want to run tests, I do it directly from the Node REPL#2022-02-1511:12Martynas MSo you develop without a REPL?#2022-02-1511:13Martynas MI'm used to have everything under one process in Java. That's why I ask. I know that in CLJS it's different.#2022-02-1511:14Martynas MSo should it be ./bin/node and that's it? And then you'd proceed to go and change the sources?#2022-02-1511:14Martynas MAnd have the REPL running just in case?#2022-02-1511:15borkdudeI do use the REPL, but for CLJS I found it too brittle to do it from my editor so I chose to simplify things. I use (require '[ns] :reload) in the REPL when I want to reload changes, and invoke test expressions or files when I want to test something. This is a spartan workflow, but it's the least time consuming for me personally.#2022-02-1511:17Martynas MShould I also add how to start-up the emacs REPL into the README? I'll make a PR for a bugfix.#2022-02-1511:18borkdudeI was just providing info, I'm not the maintainer of this project, so I'll let @ikitommi or whoever decide on that :)#2022-02-1513:27dvingoI've had success running the cljs tests in malli repo in a repl, but I augmented the shadow-cljs.edn file to load one of the app targets in a browser, and then when the tests were working in the browser I verified them for node by running ./bin/node this was from cursive though. it should also work with cider with the relevant shadow integration#2022-02-1710:34DrLjótssonIs there a way to specify that a set needs to have at least one element? [:+ ...] only seems to work for sequences#2022-02-1710:43DrLjótssonThis is what I have come up with
[:and
[:set :int]
[:fn (fn [x] (not (empty? x)))]]
#2022-02-1711:11ikitommimaybe [:set {:min 1} :int]?#2022-02-1711:37DrLjótssonThat’s it, works!#2022-02-1711:37DrLjótssonThanks#2022-02-2019:48Oliver MarksHi, hoping someone can point me in the right direction, can you extend json-transformer-provider https://github.com/metosin/reitit/blob/198cfda00d20093f3d7b3069e5e902835c396698/modules/reitit-malli/src/reitit/coercion/malli.cljc#L35 I basically want to convert a specific keys value to a keyword the malli spec is an enum of keywords and the json decoder is not converting the string to a keyword for me so I need some custom logic here I believe, but I am unsure how to inject this ?#2022-02-2214:48rovanionI'm getting data to a Reitit handler with malli coercion from a simple HTML form. One of the fields should either be a number or be left blank when the form is submitted. I want to translate that to either nil or to remove the key entirely from the map during coercion, is that possible?#2022-02-2214:50rovanionRight now coersion fails because the feild holds a string when it should hold an int, but just defining the value of the field to be [:or string? int?] means that all other functions have to deal with the possibility of that field being a non-int.#2022-02-2215:21dvingoyou could make your own schema type and then add custom coercion logic there. For example this is a custom schema for local dates:
:local-date (m/-simple-schema
{:type :local-date
:pred tu/date?
:type-properties {:gen/gen gen-date
:decode/string (fn [v]
(log/info "Decoding to local-date: " v)
(tu/date v))}})
and add that to your malli registry#2022-02-2215:23dvingoand to deal with the logic of conditionally removing the key from a map, my guess is that you could change the malli coercer:
https://github.com/metosin/reitit/blob/master/doc/coercion/malli_coercion.md#configuring-coercion
by changing the transformers#2022-02-2215:43rovanionInteresting, I like that first idea. Make a type with a name like :nilable-form-int, simple enough. Thank you!
Writing a coercer would probably be better, but I don't think I could wrap my head around it fast enough to still please the deadline.#2022-02-2215:55ikitommi@rovanion maybe:
(def NilableInt
[:maybe {:decode/string (fn [x] (when-not (str/blank? x) (mt/-string->long x)))} :int])
(m/decode
[:map
[:x NilableInt]
[:y NilableInt]]
{:x ""
:y "123"}
(mt/string-transformer))
; => {:x nil, :y 123}
(mg/sample
[:map
[:x NilableInt]
[:y NilableInt]])
;({:x -1, :y nil}
; {:x nil, :y nil}
; {:x 0, :y -1}
; {:x nil, :y nil}
; {:x 2, :y nil}
; {:x -3, :y -1}
; {:x -2, :y nil}
; {:x 3, :y 1}
; {:x nil, :y nil}
; {:x -8, :y -1})#2022-02-2216:02rovanionHah, was just writing that exact :decode/string-function in my own simple-schema. But yeah, that fits the bill!#2022-02-2216:03rovanionI really like that Clojure has functions like when-not in the standard library so I don't have to make them up in some util library.#2022-02-2216:07rovanions/functions/functions and macros/#2022-02-2217:31rovanionThis is where I'm at now:
(register! :html/nilable-int
(malli/-simple-schema
(fn [opts _]
{:type :html/nilable-int
:pred (some-fn int? nil?)
:property-pred (malli/-min-max-pred nil)
:description "An int or nil. Empty string is transformed to nil."
:type-properties {:decode/string (fn [x] (when-not (string/blank? x)
(mtransform/-string->long x)))
:gen/gen (gen/large-integer* opts)}})))
I just have to write a generator that generates nil sometimes. Sadly :gen/schema [:maybe [int? opts]] doesn't respect :min passed in opts.#2022-02-2218:12ikitommiint? is just a predicate, use :int , which is a real schema#2022-02-2218:15rovanionThe practical implications of the distinction is lost on me, I thought both were resolved to the exact same schema.
Either way it made no difference, setting :gen/schema [:maybe [:int opts]] still generated negative numbers when opts was {:min 0}.#2022-02-2218:59ikitommi(mg/sample [:any {:gen/schema [:int {:min 100}]}])
; => (101 101 101 101 102 108 105 100 111 148)#2022-02-2219:01ikitommi(mg/sample [:maybe [:any {:gen/schema [:int {:min 100}]}]])
; => (nil nil nil 102 nil nil 102 103 111 159)#2022-02-2219:19rovanionSorry, I must have failed to press C-c C-c or something after switching from int? to :int because it's working now and I don't know what else could have caused it to fail.
Thank you so much for your time.#2022-02-2219:20rovanionThis is the schema I ended up with, does both generate, transform and conform as I want it:
(register! :html/nilable-int
(malli/-simple-schema
(fn [opts _]
{:type :html/nilable-int
:pred (some-fn int? nil?)
:property-pred (malli/-min-max-pred nil)
:description "An int or nil. Empty string is transformed to nil."
:type-properties {:decode/string (fn [x] (when-not (string/blank? x)
(mtransform/-string->long x)))
:gen/schema [:maybe [:int opts]]}})))#2022-02-2220:17Oliver MarksSo been playing a bit more and built up some snippet code to show my issues, the first works as I expect the second part does not seem to transform in the same way and I can not figure out why, can any one lend some assistance ?
(def my-spec
[:map {:closed true}
[:object/type [:keyword {:json-schema/type "keyword" :string-schema/type "keyword"}]]])
(m/decode
my-spec
{:object/type "test"}
mt/json-transformer)
;; => #:object{:type :test}
How ever using this returns an error complaining that objecct/type is not a keyword, I was under the imrpression it should be converted the same as m/decode, so I am obviously missing a trick some where, tried a few things but just can not get it to decode.
(reitit.coercion.malli/create
{:transformers {:body {:default reitit.coercion.malli/default-transformer-provider
:formats {"application/json" reitit.coercion.malli/json-transformer-provider
"application/transit+json; charset=utf-8" reitit.coercion.malli/json-transformer-provider
"application/transit+json" reitit.coercion.malli/json-transformer-provider}}
:keyword {:default reitit.coercion.malli/string-transformer-provider}
:string {:default reitit.coercion.malli/string-transformer-provider}
:response {:default reitit.coercion.malli/default-transformer-provider}}})
#2022-02-2309:25olyjust some thoughts in case its related, I am using transit+json because my keys are namespaces this is so they get preserved over the wire#2022-02-2314:05lambderhello#2022-02-2314:05lambderI tried to apply this https://github.com/metosin/malli/pull/545/files#2022-02-2314:05lambderin my project#2022-02-2314:06lambderI've added zoned-date-time#2022-02-2314:06lambderin some other ns I define schema like:
(def ZDT
(mt/time-schema :zoned-date-time))
#2022-02-2314:07lambderthen in the reitit router I have malli coercion set and the endpoint has this:#2022-02-2314:08lambder:parameters
{:body
[:map
[:scrape-request
[:map
[:id int?]
[:from-date
[:map
{:registry
#:my-name-space{:zdt :zoned-date-time}}
:my-name-space/zdt]]]]]},#2022-02-2314:08lambderbut when I try to create the routes I get:#2022-02-2314:09lambderExecution error (ExceptionInfo) at reitit.exception/exception (exception.cljc:19).
:malli.core/invalid-schema {:schema :zoned-date-time}
{:schema :zoned-date-time}#2022-02-2314:09lambderany idea?#2022-02-2314:10lambdersame problem if I don't use registry#2022-02-2314:10lambder:parameters
{:body
[:map
[:scrape-request
[:map [:id int?] [:from-date :zoned-date-time]]]]},#2022-02-2314:11lambderwhat am I doing wrong?#2022-02-2314:11lambderplease?#2022-02-2314:45lambderI probably son't use the default registry correctly. Any hints please?#2022-02-2315:03pithylessInstead of :zoned-date-time try referring directly to your var ZDT#2022-02-2315:04pithylessIf you're using the default registry, you need to make sure it registers :zoned-date-time as a known schema. See:
https://github.com/metosin/malli/blob/2398df55ee806e25592fabf4d0c642ee3a2b233f/src/malli/core.cljc#L2371-L2378#2022-02-2315:21lambder@U05476190 thanks, I did small step forward. The route construction no longer fails after I defined the custom registry via :coercion
(defn service-routes []
["/api"
{:coercion (reitit.coercion.malli/create
(merge reitit.coercion.malli/default-options
{:options {:registry (merge
(m/default-schemas)
(malli-time/time-schemas))}}))#2022-02-2315:21lambderbut , on Swagger UI I'm getting:#2022-02-2315:22lambder\#2022-02-2315:24lambderthe swagger.json is:
#2022-02-2315:25lambderI 'believe it should be:
"type": "string",
"format": "date-time"
no?#2022-02-2315:27pithylessSorry, no idea; probably best to ask outside of this thread - I'm not familiar with swagger/malli integration#2022-02-2315:36lambderwill try to update https://github.com/metosin/malli/pull/545/files with#2022-02-2315:37lambdermalli offers json swagger schema integration#2022-02-2317:05lambderhere @ronny#2022-02-2316:39Ronny LøvtangenThanx @lambder, I was trying to add support for java.time.LocalDate and java.time.Instant to my project, and your post helped me.
I added the code from https://github.com/metosin/malli/pull/545/files and imported it to the registry with
(defn service-routes []
["/api"
{:coercion (reitit.coercion.malli/create
{:options {:registry (merge
(m/default-schemas)
(malli-time/time-schemas))}})
(I believe the merge with default-options is not necessary, as reitit.coercion.malli/create already does that)
I was then able to coerce a string to java.time.LocalDate by specifying the property as:
[:mydate {:description "My description"
:json-schema/example "2021-11-05"}
:local-date]
One thing I noticed, was that if I tried to send a non-parsable value, I got
Error: Wrong number of args (2) passed to: slakt-service.malli.time/->error-reporter/-report--28426
By changing
(defn ->error-reporter
[parser message]
(fn -report [value]
to
(defn ->error-reporter
[parser message]
(fn -report [value _]
I instead got the message
"Should be local-date or LocalDate"
#2022-02-2317:04lambdergreat stuff, @ronny#2022-02-2317:05lambderhave you seen the massages in the thread?#2022-02-2317:05lambderone thing I'm missing is to make it work with edn request#2022-02-2317:05lambderit works with json only#2022-02-2317:06lambderI'll do
`(fn -report [value & _]
`#2022-02-2317:06Ronny LøvtangenYes. I was also wondering about type vs format. Not an expert on json schema, but looks like you are right#2022-02-2317:06lambderjust in case some other call needs single arity#2022-02-2317:10Ronny LøvtangenI also added :json-schema/example:
:local-date {:class LocalDate :parser -string->local-date :json-schema/type :local-date :json-schema/example "2022-02-23"}
...
:json-schema/example (:json-schema/example props -name)
Then I don’t have to (but could override if I want to) provide example in my schema.#2022-02-2317:10lambderif I don't supply the example for date-time, the SwaggerUI will get one form me#2022-02-2317:17Ronny LøvtangenIf I don’t provide json-schema/example, Swagger will show “Unknown Type: local-date”.
Maybe your change to
"type": "string",
"format": "date-time"
is what makes Swagger able to provide an example#2022-02-2317:17lambderlocal-date is not corret#2022-02-2317:18lambderthere is only date and date-time#2022-02-2317:18lambdersee here https://www.baeldung.com/openapi-dates#2022-02-2317:18lambderfor local dates you probably want to adapt:#2022-02-2317:18lambdercustomDate:
type: string
pattern: '^\d{4}(0[1-9]|1[012])(0[1-9]|[12][0-9]|3[01])$'
description: Custom date
example: "20210130"#2022-02-2317:23Ronny LøvtangenIsn’t “date” a good fit for java.time.LocalDate? Format yyyy-MM-dd#2022-02-2317:23lambderit is#2022-02-2317:27Ronny LøvtangenWith
:local-date {:class LocalDate :parser -string->local-date :json-schema/type :string :json-schema/format "date"}
Swagger provides an example :thumbsup:#2022-02-2317:28lambderright#2022-02-2317:28lambderdo you have your app working with edn encoding?#2022-02-2317:29lambderdo I need to provide analog of:#2022-02-2317:29lambder{:transformers {:body {:default default-transformer-provider
:formats {"application/json" json-transformer-provider}}
:string {:default string-transformer-provider}
:response {:default default-transformer-provider
:formats {"application/json" json-transformer-provider}}}#2022-02-2317:29lambderfor edn?#2022-02-2317:33Ronny LøvtangenNo, sorry, haven't done anything to provide support for edn in our application#2022-02-2318:36lambderwhen my edn request is:#2022-02-2318:36lambder{
:scrape-request {
:id 1
:from-date #time/zoned-date-time "2022-01-01T16:14:34.447Z"
}
}#2022-02-2318:37lambderI'm getting:#2022-02-2318:37lambder{:schema "[:map {:closed true} [:scrape-request [:map {:closed true} [:id int?] [:from-date :zoned-date-time]]]]", :errors ({:path [:scrape-request :from-date], :in [:scrape-request :from-date], :schema ":zoned-date-time", :value (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z"), :message "Should be zoned-date-time or ZonedDateTime"}), :value {:scrape-request {:id 1, :from-date (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z")}}, :type :reitit.coercion/request-coercion, :coercion :malli, :in [:request :body-params], :humanized {:scrape-request {:from-date ["Should be zoned-date-time or ZonedDateTime"]}}}#2022-02-2318:37lambdersee: :from-date (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z")#2022-02-2318:37lambderhow on earth this form gets there are all?#2022-02-2318:38lambderthis is added to the registry:#2022-02-2318:38lambder:zoned-date-time #IntoSchema{:type :zoned-date-time},#2022-02-2318:39lambderusing this:#2022-02-2318:39lambder(m/-simple-schema
{:type type
:type-properties
(cond-> {:error/fn error-fn
:decode/json {:enter safe-parser}
:encode/json {:enter #(prn %)}
:json-schema/type (:json-schema/type props -name)}
(:json-schema/format props -name) (assoc :json-schema/format (:json-schema/format props -name)))
:pred pred})#2022-02-2318:40lambderthe `safe-parser` is:
#2022-02-2318:40lambderprovided by :#2022-02-2318:40lambder(defn -string->zoned-date-time [x]
(ZonedDateTime/parse x))
#2022-02-2318:40lambderso this :from-date (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z") looks a lot like (ZonedDateTime/parse x)#2022-02-2318:41lambderbut instead if being invoked is it passed a a value. How did it happen ?#2022-02-2319:15lambderfunny thing the value to decoder is coercion.malli is already wrapped with something like this:#2022-02-2319:16lambder#2022-02-2319:17lambderthis happens in reitit.coercion by transforming the request to the value#2022-02-2319:17lambder#2022-02-2319:36juhoteperi@lambder those decode/json transformations aren't used by coercion here (no coercion really needed for these values) when you are using EDN. As EDN can already represent types, the coercion doesn't do anything.
If you have *data-reader* registered for the #time/zoned-date-time tag (or provided :readers option to muuntaja edn format) the value should be correct and the problem must be with the schema validation (the predicate).#2022-02-2319:38lambderhey @juhoteperi no, I don't have the readers supplied to muuntaja sepcially#2022-02-2319:38lambderI got `
:muuntaja formats/instance
#2022-02-2319:39lambderthat instance is:#2022-02-2319:39lambder(def instance
(m/create
(-> m/default-options
(update-in
[:formats "application/transit+json" :decoder-opts]
(partial merge time/time-deserialization-handlers))
(update-in
[:formats "application/transit+json" :encoder-opts]
(partial merge time/time-serialization-handlers)))))#2022-02-2319:39juhoteperiMuuntaja will use *data-readers*, which is already provided by Clojure if you have the readers in a data_readers.clj file somewhere#2022-02-2319:40lambderI do in my core ns `
(time-literals.read-write/print-time-literals-clj!)
#2022-02-2319:40juhoteperiBut hmm, it could be problem with those data readers, because :value (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z") seems really strange. Looks like the value isn't the ZonedDateTime instance, but the Clojure form calling that ZonedDateTime/parse method#2022-02-2319:40lambderbut not setting it for read#2022-02-2319:40lambderwill get back to it in about 1h30m#2022-02-2319:41lambderneed to run now#2022-02-2319:41juhoteperi> The library includes the magic file data_readers.cljc which Clojure and the Clojurescript compilers() will look for.
#2022-02-2319:41lambderthanks for everything#2022-02-2319:41juhoteperiSo just including time-literals in the classpath should add data-readers for those tags#2022-02-2319:44juhoteperiIf data-readers are working correctly, if you eval in the REPL {:foo #time/zoned-date-time "2022-01-01T16:14:34.447Z"} you should get the same map back, the tagged value is read to ZonedDateTime instance, and because you have the print method registered, it should be printed out the same way.#2022-02-2319:55lambderIn need to add time readers to end/read-string option for it to work#2022-02-2319:56lambderSo I guess they are not added to data_readers, but I'll dbl check#2022-02-2323:02lambder@juhoteperi this is just black magic to me#2022-02-2323:02lambder#2022-02-2406:41henryw374@lambder above issue fixed on the latest release - see https://github.com/henryw374/time-literals readme for details.
tl;dr it was basically a workaround for a problem with Clojurescript that has taken some time to get fixed - see https://clojure.atlassian.net/browse/CLJS-3294#2022-02-2408:47lambder@henryw374 Thank you !#2022-02-2409:40lambderthis worked like a charm 😄#2022-02-2411:01rovanion@ikitommi I wrote a version of -min-max-pred that can deal with nil being passed to it as I could not figure out an f that could transform nil into a number that would make sense min and max to apply to. If you think you have use for it anywhere in Malli, here it is:
(defn -nilable-min-max-pred [f nilable]
(fn [{:keys [min max]}]
(cond
(not (or min max)) nil
(and min max f nilable) (fn [x] (if (nil? x) true (let [size (f x)] (<= min size max))))
(and min max f) (fn [x] (let [size (f x)] (<= min size max)))
(and min max nilable) (fn [x] (if (nil? x) true (<= min x max)))
(and min max) (fn [x] (<= min x max))
(and min f nilable) (fn [x] (if (nil? x) true (<= min (f x))))
(and min f) (fn [x] (<= min (f x)))
(and min nilable) (fn [x] (if (nil? x) true (<= min x)))
min (fn [x] (<= min x))
(and max f nilable) (fn [x] (if (nil? x) true (<= (f x) max)))
(and max f) (fn [x] (<= (f x) max))
(and max nilable) (fn [x] (if (nil? x) true (<= x max)))
max (fn [x] (<= x max)))))
And its use in my code is just:
(register! :html/nilable-int
(malli/-simple-schema
(fn [opts _]
{:type :html/nilable-int
:pred (some-fn int? nil?)
:property-pred (su/-nilable-min-max-pred nil true)
:description "An int or nil. Empty string is transformed to nil."
:type-properties {:decode/string (fn [x] (when-not (string/blank? x)
(mtransform/-string->long x)))
:gen/schema [:maybe [:int opts]]}})))#2022-02-2513:34Oliver Marksso still struggling with transformers not being applied, I am looking at the request chain I can see that :body-params is populated with a namespaced map, how ever in the next steps I get this
+:body-params {:object/type "test"}}
--- :request---
nil
--- :request :reitit.ring.coercion/coerce-exceptions ---
{}
--- :request :reitit.ring.coercion/coerce-response ---
{}
--- :response---
{:body {:coercion :malli,
:errors ({:in [],
:message "invalid type",
:path [],
Seems a bit strange that the :in would be blank, I am getting this from a test which I am using to help me figure out my issue also invalid type is not over descriptive in this context seems, any tricks to getting more info from malli on what its doing, or does this help any one give me some suggestions ?#2022-02-2513:50Oliver MarksI would also be intrested in a bit more info on this map, https://github.com/metosin/reitit/blob/198cfda00d20093f3d7b3069e5e902835c396698/modules/reitit-malli/src/reitit/coercion/malli.cljc#L110 are there any docs on what going on is :transformers the function that gets applied to produce the body-params ie is it this step which should be converting my strings to keywords based on the schema ?#2022-02-2516:34Grant HornerRunning into a weird issue: I’m getting the following report on a function call in my test suite:
{:in [1 0 :foo-id],
:message "should be a positive int",
:path [1 0 :foo-id],
:schema pos-int?,
:value 9218}
where, according to the report, the value is in fact a positive integer. The function is expecting a
[:sequential [:map [:foo-id pos-int?] [:name string?]]]
as an input, and is receiving
({:name "Foo Inc", :foo-id 9218, :type "security-id", :value 9218})
as the arg. This correctly passes the call to malli.core/validate , but is failing in my test suite for some reason#2022-02-2516:47Grant Hornerfalse alarm. the value is actually a biginteger. maybe the error message could be improved in these cases? perhaps including the value’s type?#2022-02-2519:35Oliver MarksI may have had a bit of progress, I seem to be able to make it work using application/json in place of application/transit+json I was using the later as cljs-ajax drops the namespace in the keys which I was hoping to avoid. could this be a case of the flow is different for application/transit+json I did add it to the coercsion map like below.
{:transformers {:body {:default reitit.coercion.malli/default-transformer-provider
:formats {"application/json" reitit.coercion.malli/json-transformer-provider
"application/transit+json" reitit.coercion.malli/json-transformer-provider}}#2022-02-2600:23sound2gdhi clojurians~ I'm stucking on decoding json from string with malli#2022-02-2600:23sound2gd(require '[malli.core :as m])
(require '[malli.transform :as mt])
(m/decode [:map
[:x :string]
[:y :int]]
"{\"x\": \"aa\",\"y\": 3}"
(mt/transformer
mt/json-transformer))
;; => "{\"x\": \"aa\",\"y\": 3}"#2022-02-2606:16Ben SlessJson transformers assume the data is already deserialized#2022-02-2606:17Ben SlessYou need a library like jsonista for that#2022-02-2614:35sound2gdthanks. finally work out like this#2022-02-2614:35sound2gd(m/decode [:string {:decode/string #(json/decode % true)
:encode/string json/encode}]
"{\"x\": \"aa\",\"y\": 3}"
(mt/transformer
mt/strip-extra-keys-transformer
mt/default-value-transformer
mt/string-transformer))#2022-02-2815:04rovanionGiven a Malli map schema like [:map [:key {:optional true} :int]], how do I get at the properties assigned the keyword :key in the map when writing a walker? In this example, these are {:optional true}.
As you can see in the below example the walker visits all the "values" of the map, in this case just the :int schema, and then out to the :map schema. Is there any way to get to the inbetween step, the "schema" and properties of :key?
(malli/walk
[:map [:key {:optional true} :int]]
(fn [schema path children options]
(println "Schema:" schema)
(println "Path:" path)
(println "Children:" children)
(println "Options:" options)
(println "Properties:" (malli/properties schema))
(println))) => nil
;; Schema: :int
;; Path: [:key]
;; Children: nil
;; Options: nil
;; Properties: nil
;;
;; Schema: [:map [:key {:optional true} :int]]
;; Path: []
;; Children: [[:key {:optional true} nil]]
;; Options: nil
;; Properties: nil
#2022-02-2815:29ikitommiWould this help? https://cljdoc.org/d/metosin/malli/0.8.3/doc/tips#walking-schema-and-entry-properties#2022-02-2816:22rovanionAh, okey, so you need to pass {::malli/walk-entry-vals true}.#2022-02-2816:22rovanionThanks.#2022-02-2820:00Oliver MarksSo setup a test case from the example project, https://github.com/olymk2/reitit/commit/6fd0ee5b6d1553cfc25227836b8a3a126b77b5d5 I have added a single param to the malli spec if you just submit the defaults it sends "january" and errors because its not a key my understanding is this should be decoded by the transformer ? is this a bug or something I am not understanding ?#2022-03-0101:01vemvIs there a supported way to introspect that the type of the following schema is string??
[:fn {:description "An ID"}
string?]
malli.core/type returns :fn for it, which is not as useful.
...`(m/type string?)` does return string? (as a symbol), so I'd say that my expectation is reasonable#2022-03-0101:36vemvI had some luck with considering this vector to be "just data" but yeah... an official helper would be reassuring#2022-03-0104:32Ben SlessMy unofficial hot take is fn specs are evil for this very reason#2022-03-0117:15CaseyIs it not possible to pass an options map (containing a registry) to malli.util/update-in ? The other functions in that ns take options map, but this one doesn't appear too. update also lacks an options arg. I can't use a local registry because I've a non-map (primitive) registry.#2022-03-0213:19DenisMcHi, I would like to disallow null bytes in :string fields in my malli schema because Postgres can’t handle them. What is the cleanest way to add this sort of validation? Thanks in advance.#2022-03-0213:43DenisMcJust looking at this further: I’m running into this null byte generator issue because I need some of my strings to be non-empty, so I’m setting :min (and :max) parameters in Malli, like so:
[:my-param [:string {:min 1 :max 5}]]
I see that the malli.generator/-string-gen code looks like this:
(defn- -string-gen [schema options]
(let [{:keys [min max]} (-min-max schema options)]
(cond
(and min (= min max)) (gen/fmap str/join (gen/vector gen/char min))
(and min max) (gen/fmap str/join (gen/vector gen/char min max))
min (gen/fmap str/join (gen/vector gen/char min (* 2 min)))
max (gen/fmap str/join (gen/vector gen/char 0 max))
:else gen/string-alphanumeric)))
..so the generator generates alphanumeric strings if the min or max length are not set, but generates random characters (including non-alphanumeric characters) if the length is set. I’m wondering what the logic is for this? Why does the string length determine what sort of characters are included in the randomly generated strings?#2022-03-0412:05ingesolHi! I’m trying to get malli 0.8.4 to generate clj-kondo config for my browser CLJS project, trying this code
(ns mynamespace
(:require-macros [malli.dev.cljs :as dev]))
(defn test-fn
"return y when x is larger than 5"
{:malli/schema [:=>
[:cat :int :keyword]
[:maybe :keyword]]}
[x y]
(when (< 5 x)
y))
(dev/start! nil)
(test-fn 1 2)
Instrumenting seems to work and I’m getting the expected validation error in my browser console, but I’m seeing no clj-kondo config being generated for my function schema, only seeing this in .clj-kondo/metosin/malli-types:
{:linters {:unresolved-symbol {:exclude [(malli.core/=>)]}}}#2022-03-0716:16dvingoit looks like cljs support broke during a refactor of how function schemas are stored (there is now a map for :clj and one for :cljs)
so the kondo data is looking up :clj even in :cljs code:
https://github.com/metosin/malli/blob/002d5cbc724e83f07f126d66b19df022859f1cbf/src/malli/clj_kondo.cljc#L179
https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2385
I should have some time over the next few days to send a pr to address this#2022-03-0717:25ingesolRight, that’s what I suspected too, since the CLJS support was pretty fresh. Thank you so much for looking into it!#2022-03-0416:02Martynas MHey. Is there a way to generate a value from a predefined list of values?
Or maybe it's possible to "remember" previously generated values so that they would be able to be reused to produce random connections (as in DB sense)?#2022-03-0716:46ikitommimaybe:
(mg/sample [:any {:gen/elements [1 2 3]}])
; => (1 1 3 2 1 1 2 1 1 1)#2022-03-0416:21mafcocincoSimple newbie question: For malli.generator/generate is there a way to force malli to generate all fields, including optional?#2022-03-0716:45ikitommiat the moment, no. But would be 1-2 extra lines of code:
• new option :mg/generate-optional-values
• read it here https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L90 - if true always, if false never, default to nil (current)
• add a test
• document int README#2022-03-0518:48match37Using malli 0.8.4, got below error, what did I miss? Appreciate your help!
user=> (require '[malli.core :as m ])
nil
user=> (def s (->> [:union
[:map [:x :string]]
[:schema [:map [:x :int]]]]))
#'user/s
user=> (m/form s)
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
:malli.core/invalid-schema {:schema :union}
user=> (m/validate s {:x "x"})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
:malli.core/invalid-schema {:schema :union}
#2022-03-0604:51match37answer my own question, I need register some predefined schemas.
{:registry (merge (mu/schemas) (m/default-schemas))}#2022-03-0714:04henrikIs it possible to spec multi arity functions with the experimental Plumatic-style schema? https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-inline-schemas#2022-03-0714:08ikitommisure:
(mx/defn plus
([x :- :int] x)
([x :- :int, y :- :int] (+ x y))
([x :- :int, y :- :int, & zs :- [:* :int]] (apply + x y zs)))
#2022-03-0714:09henrikDoh, naturally, thank you.#2022-03-0714:09ikitommi👍#2022-03-0813:14henrikMy head was not screwed on right yesterday. What I meant to ask was, how do I scope a particular return spec to a particular arity.
In e.g. Snoop, this is possible since the return spec is part of the arity. In Plumatic, I'm unsure what the syntax is.#2022-03-0815:05ikitommithere is no such thing in the Plumatic Syntax. Thought of adding that after the args-vector, but then it would not be Plumatic Syntax anymore.#2022-03-0815:05ikitommiI propose to ask this from the plumatic schema gang via an issue, happy to incorporate whatever they think is good, could add my 2 cents in the conversation there#2022-03-0815:29ikitommi… and you can do that with non-plumatic syntax already.#2022-03-0815:30ikitommi(defn f1
"doc"
(^{:malli/schema [:=> [:cat :int] :int]} [x] (inc x))
(^{:malli/schema [:=> [:cat :int :int] :int]} [x y] (+ x y)))
or
(defn f2
"doc"
{:malli/schema [:function
[:=> [:cat :int] :int]
[:=> [:cat :int :int] :int]]}
([x] (inc x))
([x y] (+ x y)))#2022-03-0908:26henrikI think it would be good to alias the Plumatic macro name to something like >defn as well. Partly because it seems to be a bit of a convention, partly because it makes it possible to refer it without breaking the equivalent in clojure.core.#2022-03-0908:28henrikRegarding arity-specific return specs; to me it makes sense to tack them on to the param vectors. It might even be possible to combine the two: a spec that runs regardless of the arity, and a spec that runs per arity.#2022-03-0909:44ikitommiPersonally, I really don’t like the >defns. clojure.test is the only ns that I require macros without namespace alias. mx/defn tells where it is coming from.#2022-03-0909:45ikitommi(mx/defn plus :- :int ;; the default, masked by 2 arities?
([x :- :int] :- :int x)
([x :- :int, y :- :int] (+ x y))
([x :- :int, y :- :int, & zs :- [:* :int]] :- :int (apply + x y zs)))
#2022-03-0909:47ikitommistarting to be quite verbose. Have asked if Cursive could grey out the schema hints so it would be move visbile#2022-03-0911:10henrik> Personally, I really don’t like the >defns
Fair enough. I tend to lean towards referring, when it's canonical or conventional (we decide on one >defn, and overloading it is an error), and >defn has become something of a naming convention among function spec libraries.
> verbose
Agreed, and it's also not that easy to make a formatter understand the syntax to begin with. The separate vector approach of Snoop/Guardrails etc. has the advantage of registering as a conventional function in terms of arguments fed to it. By contrast, Plumatic seems like it requires quite a bit of interpretation of syntax, especially when extended in this way. It also has advantages, of course.#2022-03-0903:53Ryan TateHello, Malli looks very cool.
Is it possible in Malli to retain values generated during validation like spec/conform and spec/conformer? Or if not maybe you can think of some Malli-ish way to solve this:
I have some expensive validations, for example I accept an xpath as input and validate by compiling it. I don't want to throw the resulting value away and have to re-generate it later. So I transform (with s/conformer) the xpath string to a map with a compiled version (object) of it during validation and this map is the result of validation (with s/conform):
(require '[clj-xpath.core :as xpath])
(defn expr-or-nil
[maybe-expr]
(try
(xpath/xp:compile maybe-expr)
(catch org.xml.sax.SAXException e)
(catch javax.xml.xpath.XPathException e)))
(s/def ::xpath (s/and string?
(s/conformer (fn [value] {:value value
:expr (expr-or-nil value)})
:value)
#(:expr %)))
(s/conform ::xpath "//a/@href")
;; {:value "//a/@href",
;; :expr
;; #object[org.apache.xpath.jaxp.XPathExpressionImpl 0x1aa56eab "
I do similar things when I accept regex strings as input - I validate by compiling them with re-pattern in a conformer then I save the output from s/conform for use later.
So is it possible to do something like this with Malli? The :decode/math examples in the Value Transformation section of the README.md (https://github.com/metosin/malli#value-transformation) seem possibly close? But it's not clear to me if decode validates.
Maybe Malli is trying to keep validation in a different stream from transformation?
(I admit I don't understand the transformation section of the docs very well, particularly when it comes to doing custom functions as opposed to the pre-canned transformers for json and so forth.)
Kiitos/thanks for any help#2022-03-0906:32ikitommi@ryantate you have two models there: a string and a parsed map. With malli, you should describe either the string (source) or the target (a map). If the map presentation is what you expect, describe that and add a decoder from string->it. A naive impl:
(defn expr-or-nil [maybe-expr]
(when (= maybe-expr "//a/@href") identity))
(defn decoder [value]
{:value value
:expr (expr-or-nil value)})
(def Xpath
(m/schema
[:map {:decode/string decoder}
[:value :string]
[:expr fn?]]))
(defn coerce [value]
(let [decoded (m/decode Xpath value (mt/string-transformer))]
(if-not (m/validate Xpath decoded)
::invalid
decoded)))
(coerce "//a/@href")
; => {:value "//a/@href", :expr #object[clojure.core$identity 0x2c879e65 "#2022-03-0906:35ikitommibut yes, the transform and validation are separated. There might be use cases where this is not good, but for most cases, it’s easy to compose the two using public apis. If/when someone finds out a need for one-pass-do-it-all &/ stateful transformers, would like to hear those 😎#2022-03-0913:38Ryan Tate@ikitommi Ah thanks for the comprehensive reply!! 👍
string -> compiled-xpath is the goal, the map is just there to allow s/unform which is not essential.
In my case I would probably just accept the cost of two passes because I need the xpath compilation in order to validate the string) and also the xpath string needs to be composed into a more complicated validation so I can just do one m/validate call on the larger structure (which can have many xpath strings). So I think the coerce function would be tricky to use. I could just validate all xpath as string in the larger value and then m/validate and m/parse the larger value and then walk it and call coerce on each xpath item but I would have to do that also for regex and anything else that gets compiled - becomes basically building a new validation layer which defeats the point a bit.
I did a benchmark and compiling xpath is only about 2 μs and regex is even less so I will just put those checks in the schema so it is part of a compostable validation and then do it again in m/decode. It is a performance hit but a small one, hopefully not too bad 😅#2022-03-0913:55ikitommioh, if you want to validate the big thing, you could do schema for string, and then m/encode it to the parsed format. there is no easy way to customize m/parse per schema atm.#2022-03-0913:56ikitommifor performance, you should use m/validator & m/decoder / m/encoder for MUCH better perf.#2022-03-0913:58ikitommi;; pure functions, can be cached
(def validate (m/validator Xpath))
(def decode (m/decoder Xpath (mt/string-transformer)))
(defn coercer [value]
(let [decoded (decode value)]
(if-not (validate decoded)
::invalid
decoded)))#2022-03-0918:18Ryan TateIssue with :orn schema inside decode.
Any idea why this works:
(m/decode [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}] "foo" mt/string-transformer)
;;#"foo"
But this does not?:
(m/decode [:orn [:v vector?] [:re-string [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)
;;"foo"
Is decode meant to work with :orn, :catn, :or, :and etc?#2022-03-0918:51Ryan TateActually I think I have mixed up encode and decode.
I found this which has helpful definition of differences (maybe later I can make a PR to merge this table into README.md?):
https://cljdoc.org/d/metosin/malli/0.8.4/doc/value-transformation
decode is for making values valid, encode is to transform valid values into something else.
decode will revert value if it does not match the schema per https://clojurians.slack.com/archives/CLDK6MFMK/p1615225531254000
I rewrote this way which works:
(m/encode [:orn [:v vector?] [:re-string [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)
;;#"foo"#2022-03-1006:48ikitommiDoc PR most welcome#2022-03-1006:49ikitommiAlso, a transformation debugger would be nice (and not hard to implement?): would emit which steps are run, in which order and how they change / not the value.#2022-03-0919:13Ryan TateCan I combine parse and encode functionality using a single schema?
I can parse:
(def re-schema [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}])
(def re-w-opts-schema [:and vector? [:catn [:pattern re-schema] [:opts [:* [:enum :i :s :u]]]]])
(def re-maybe-w-opts [:orn [:pattern re-schema] [:pattern-w-opts re-w-opts-schema]])
(m/parse re-maybe-w-opts ["foo" :i])
;;[:pattern-w-opts {:pattern "foo", :opts [:i]}]
I can encode:
(m/encode re-maybe-w-opts ["foo" :i] mt/string-transformer)
;;[#"foo" :i]
How to do both and get
[:pattern-w-opts {:pattern #"foo", :opts [:i]}]
?#2022-03-1007:00ikitommicurrently, you can’t combine those and there is no schema property-based extension for parsers, so you could plug in custom logic to parsing. I think you could cal m/parse in a custom encoder :leaveyourself?#2022-03-1007:03ikitommioh, the branching happens at parse, would not work that way. should add hook to parsing to allow custom steps or somehow mix the two (encode & parse). ideas welcome#2022-03-1415:31Ryan TateThanks for the reply, going to work on this for a bit. Parsing is more important for my use case than other transforms. Will come back if I have API extension ideas. Kiitos 👍#2022-03-1921:37Phil JacksonI came to ask exactly the same thing. @ryantate can you let me know if you come up with something?#2022-03-1013:40CélioMaybe I’m doing it wrong, but is there a reason why transformers don’t handle enums?#2022-03-1013:40CélioAn example with malli.transform/json-transformer:
(require '[malli.core :as m])
(require '[malli.transform :as mt])
(def schema [:enum {:encode name :decode keyword} :a :b :c])
(m/encode schema :x mt/json-transformer)
;; => :x
(m/decode schema "x" mt/json-transformer)
;; => "x"
Then I created a custom json transformer based on malli.transform/json-transformer where it adds codecs for :enum:
(defn my-json-transformer
...
(mt/transformer
{:name :json
:decoders (-> (mt/-json-decoders)
...
(assoc :enum mt/-string->keyword))
:encoders (-> (mt/-json-encoders)
(assoc :enum m/-keyword->string))}))
(m/encode schema :x my-json-transformer)
;; => "x"
(m/decode schema "x" my-json-transformer)
;; => :x#2022-03-1014:09Ben SlessTry decode/json ?#2022-03-1016:36CélioThanks, that works.#2022-03-1021:41DiegoHello everyone. What’s the best way to use malli for generative testing?#2022-03-1212:17Felipeif I’m not mistaken malli’s generators are compatible with clojure.test.check, which is what I’d use https://github.com/clojure/test.check#2022-03-1218:07DiegoThanks @UA2U3KW0L I’ll try that.#2022-03-1211:17Adam Helins@andres.rodriguezhn Edit: sorry, I was struggling with a situation similar to https://github.com/seancorfield/build-clj/issues/7 regarding tools.build and setting a custom registry. I eventually managed, thanks for raising attention on this :thumbsup:#2022-03-1309:08dharriganWhich way is the direction going re: vector vs map syntax?#2022-03-1309:08dharriganIs map the way forward?#2022-03-1311:16Ben SlessI'm guessing vector syntax will never go away due to backwards compatibility#2022-03-1314:43valtteriYou can pick whichever you prefer.#2022-03-1314:44valtteriIf I recall correctly, Tommi mentioned that map syntax is basically sugar over vector syntax#2022-03-1314:55valtteriAch, sorry it was the other way around. Map syntax is closer to the internal syntax.#2022-03-1314:57dharrigan:thumbsup:#2022-03-1413:51Martynas MHey.
What function should I use to parse a string into a number?
Let's say I have this schema:
(m/parse :int 15)
My initial assumption was that the parse would do the string parsing. But it... doesn't work at all.
How do I make this return 15 :
(m/parse :int "15")
Is this the right function to use?#2022-03-1414:06ikitommihi, it’s value transformation in malli, here are the docs: https://github.com/metosin/malli#value-transformation#2022-03-1415:46Martynas MThanks. That works.#2022-03-1415:47Martynas MWhy does parse exist then?#2022-03-1416:40ikitommiit's structural parsing, please read the part from the README. Malli itself uses it for parsing Clojure destructuring syntax to infer schemas from source code.#2022-03-1417:21Ryan TateIs it possible to combine :and with :cat? In spec this would be "`&`" (ampersand).
I need to check each key with :cat and then the two keys together via a special fn.
This works without the :and but once I add the :and and the :fn it fails:
(m/validate [:* [:and [:cat :symbol :string] [:fn (fn [[key value]] true)]]] '(regex "foo"))
Ultimate form is going to be like
'(regex "foo" xpath "bar" ...)
and I want :fn to check each pair.
Thanks for any tips.#2022-03-1417:36Ryan TateAck I don't think it will work anyway since the :cat does not produce the values for the next predicate as in spec. So even if I could make :and work it's not gonna give key/value to :fn. Hmm. Not sure how I can accomplish this. Maybe I can parse inside the validation#2022-03-1417:43Ryan TateOK I solved it like this, the special function is the inner one:
(m/validate [:fn (fn [seqn] (every? (fn [[key value]] true) (m/parse [:* [:cat :symbol :string]] seqn)))] '(regex "foo" xpath "bar"))
👍
If anyone has a better idea just post here 🙃#2022-03-1417:49Ryan TateIn order to make it work and composable with m/parse I think the schema will need to repeat the seqex. So parse will run 2x. Probably will still be faster than spec:
[:and [:fn (fn [seqn] (every? (fn [[key value]] true) (m/parse [:* [:cat :symbol :string]] seqn)))] [:* [:cat :symbol :string]]]
#2022-03-1417:59Ryan TateWould still be nice if :and worked with cat though because then I could match multiple schemas across the same sequence. hmm#2022-03-1707:45rovanionDoes Malli happen to have a shorthand for making a map out of already defined value-specs like clojure.spec's (s/keys :req-un [:thing/omabob])?#2022-03-1707:52ikitommi[:map {:registry {"kikka" :int, "kukka" :string}} "kikka" "kukka"]#2022-03-1707:53ikitommie.g. if you have a schema in a registry, you can use the name instead of full entry definition.#2022-03-1708:30rovanionRight, thanks. And if I wanted to make the key in the map unqualified while the entry in the registry is qualified by a namespace, any way around that?#2022-03-1708:31rovanionWell, [:key :ns/key] works of course.#2022-03-1710:22ikitommiyes, and you can make add entry props too: [:map [::id {:optional true}]].#2022-03-1710:23ikitomminot sure was adding all that sugar a good idea, made the parser more complex - and slower#2022-03-1713:52rovanionKey/entry props I do use, so I appreciate them.#2022-03-1713:57rovanionIs it possible to add properties to an already existing custom type?
(def registry
(atom {}))
(defn register! [type ?schema]
(swap! registry assoc type ?schema))
;; Combine the default registry with our own mutable registry.
(mreg/set-default-registry!
(mreg/composite-registry
(mreg/fast-registry (malli/default-schemas))
(mreg/mutable-registry registry)))
(register! :non-empty-string [:string {:min 1}])
(malli/validate :non-empty-string "Bengt")
;; => true
(malli/validate [:map [:namn :non-empty-string]] {:namn "Bengt"})
;; => true
(malli/validate [:map [:namn [:non-empty-string {:max 2}]]] {:namn "Bengt"})
;; Throws:
;; Execution error (IllegalArgumentException) at malli.core/eval15223$fn$G (core.cljc:22).
;; No implementation of method: :-into-schema of protocol: #'malli.core/IntoSchema found for class: clojure\
.lang.PersistentVector
;; While this works:
(malli/validate [:map [:namn [:string {:max 2}]]] {:namn "Bengt"})
#2022-03-1716:58ikitommicurrently no, but would be a ~1 line change, haven’t thought of that use case… please write an issue,#2022-03-1902:39vinurshello, i define a date schema, validate it is false, what the correct format is ?
(def birthday
[:fn {:error/message "wrong birthday format" :description "birthday"}
(partial instance? java.time.LocalDate)])
(m/validate birthday "1987-09-06") => false
#2022-03-1908:36Ben SlessYou need to add a decoder and decode before validation#2022-03-2018:50ikitommi((partial instance? java.time.LocalDate) "1987-09-06")
; => false
#2022-03-1919:55esp1Quick question: what is the purpose of having equivalent predicate and type schemas, e.g. int? and :int? Is it simply for convenience?
Also, i'm noticing that when I validate using :vector I get an error, while when I use vector? it works:
(m/validate :vector [1 2 3])
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
; :malli.core/child-error {:type :vector, :properties nil, :children nil, :min 1, :max 1}
(m/validate vector? [1 2 3])
true
#2022-03-1919:56esp1this is with malli 0.8.4#2022-03-2018:48ikitommi> Is it simply for convenience?
yes, for people coming from spec. the predicates will be made optional in the future,#2022-03-2018:49ikitommicurrently, the :vector expects one child. “vector of anything” would be [:vector :any]#2022-03-2101:17steveb8nAlso useful if persisting specs to a db which is sometimes useful#2022-03-2200:28esp1cool, thanks for the explanation!#2022-03-2116:06Alex SkyHello! Is it possible to use malli with compojure + swagger? Couldn’t find any examples. I can only see an option to implement the protocol Coercion#2022-03-2116:32ikitommiSadly no. Reitit has built-in support for malli#2022-03-2205:45Alex SkyIs it related to some kind of restrictions? I’m a little surprised since compojure-api as well retitit and malli are created by metosin :)#2022-03-2206:11bedersI have a hack to turn a malli spec into a Schema, if that helps#2022-03-2206:14Alex Sky> I have a hack to turn a malli spec into a Schema, if that helps
Yes, could you show me an example?#2022-03-2206:20bedersI haven’t tested that on a compojure-api route yet, but here’s the gist of it:
(require '[schema.core :as schema]
'[schema.spec.core :as schema.spec]
'[schema.spec.leaf :as leaf])
(defn malli->schema [malli-schema]
(reify schema/Schema
(spec [this]
(leaf/leaf-spec
(schema.spec/precondition this
(fn [data]
(m/validate malli-schema data))
(fn [data]
(me/humanize (m/explain malli-schema data))))))
(explain [this]
(str "Malli->Schema:" malli-schema))))#2022-03-2206:21bedersbut this allows you to use schema like this:
(schema/check
(malli->schema [:map [:foo int?] [:bar keyword?]])
{:foo 1 :bar 2})
=> (not {:bar ["should be a keyword"]})
(schema/check
(malli->schema [:map [:foo int?] [:bar keyword?]])
{:foo 1 :bar :humbug})#2022-03-2206:23bedersnot perfect, and I’m about to use this in :body and :path-params and see if that works#2022-03-2206:56Alex SkyThanks!#2022-03-2122:42Godwin Kois it possible to get the current index of a vector spec, so that I can have different validation logic for the exact nth element?#2022-03-2206:40ikitommino, but use can use sequence schemas for that#2022-03-2301:02Godwin Kook, but it’s a bit clumsy if the validation of a specific item share most but just varies a little bit from the other…… thx a lot for your prompt respond anyway :man-bowing::skin-tone-2:#2022-03-2305:32ikitommiyou can also compose with [:and [:vector :int] [:fn (fn [v] (= (nth v 5) 42))]]#2022-03-2305:56Godwin Kogot it, i.e. raise one level to apply function spec on the entire vector :thinking_face:#2022-03-2305:59Godwin Koin our use case, we have vector of maps, using function spec to replace the entire map schema still not that straight forward or desirable…… 😅#2022-03-2212:19Nikolas PafitisI have this schema
(def ResourceIdentifier
[:schema {:registry
{::resource-key :keyword
::path-param [:or :string :int]
::query-params map?
::resource-ident [:cat
[:+ [:or
[:ref ::path-param]
[:ref ::resource-key]]]
[:? [:ref ::query-params]]]}}
::resource-ident])
and i get a :malli.core/potentially-recursive-seqex although I don't see how it's recursive.#2022-03-2212:30ikitommiyou should take away the :ref wrapping. :ref is potentially recursive. Using a plain reference value (e.g. ::path-param) should work ok.#2022-03-2310:36Nikolas Pafitis@U055NJ5CC I see, thanks alot.#2022-03-2212:25Nikolas PafitisThe equivalent in clojure.spec works fine
(s/def ::resource-key keyword?)
(s/def ::path-param (s/or :string string? :int int?))
(s/def ::query-params map?)
(s/def ::resource-ident (s/cat :path (s/+ (s/or :resource-key ::resource-key
:path-param ::path-param))
:query-params (s/? ::query-params)))#2022-03-2311:35CarloI'm looking into trying malli for my next project; is there some generative testing facility (I can't find it). How do you do generative testing?#2022-03-2311:36dharrigan#2022-03-2311:41Carlothanks @dharrigan, so to check that a function respects the malli spec you have your own thing that generates arbitrary input data and then calls the function?#2022-03-2311:42ikitommiwhat about: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defn-checking#2022-03-2311:43ikitommiIf there are some evident tools missing (I think there are), please tell 🙂#2022-03-2311:43Carloyes thanks, I don't know how I missed this file while googling 😍#2022-03-2311:44ikitommialso, you can get a test.check generator out of a schema with (malli.generator/generator my-schema).#2022-03-2401:32Carlotwo follow-up questions: is there a built-in way to (mi/check) only one function? Is there a way to pretty-print the result of (mi/check)?#2022-03-2410:14CarloTo check just one function from emacs, I ended up doing:
(defun med/cider-eval-on-top-level-form (fn-str)
(let ((quoted-defn (concat "'" (cider-defun-at-point))))
(cider-interactive-eval (concat "(" fn-str " " quoted-defn ")"))))
(defun malli-check-this ()
(interactive)
(med/cider-eval-on-top-level-form
"#(mi/check {:filters [(mi/-filter-var #{(resolve (second %))})]})"))#2022-03-2412:38CarloAbout the (mi/check) pretty-printing issue, I ended up coding my visualization for #portal that looks like this (it happens when I issue the mi/check command, that I bound to a key):#2022-03-2413:41ikitommiLook great! There is malli.dev.pretty for pretty printing things. Could add a handler for check there#2022-03-2413:50CarloThank you! Mostly, the question I had while doing this was, is there a malli spec for the kind of errors that are generated by (mi/check)?
I temporarily shimmed it as:
(def check-error
(m/schema
[:map-of symbol? [:map
[:errors [:sequential
[:map
[:check [:map
[:malli.generator/explain-output :any]]]]]]]]))
but maybe you have something more complete. Btw, I'm really liking malli, it's really well designed! 😍#2022-03-2414:22ikitommithanks! there is no schema for it, but could be, PR welcome.#2022-03-2416:25rovanionIs the following untrue because what's compared is just the addresses in the references?
(deftest optionalize-db-generated-keys
(= (malli.util/optional-keys [:map [:key [:int]]])
(malli/schema [:map [:key {:optional true} [:int]]])))#2022-03-2416:54ikitommiRecall there is mu/equals, using just form checking. Could be improved...#2022-03-2416:42CarloI asked this quite some time ago, and IIRC at the time it wasn't possible: what's the way of writing a spec for the function (defn add [x y] ...) so that we check that the result is bigger than both x and y?#2022-03-2416:53Carlooh yeah I found a relevant issue https://github.com/metosin/malli/issues/608#2022-03-2417:11Carlo@U055NJ5CC I see that https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2443-L2445 is the place in which the function is instrumented, and that's probably where a check for the new :fn key of :=> should go. But what else should be changed?#2022-03-2501:21ambrosebsI recently added static type checking for malli schemas to Typed Clojure, here's a writeup on how to do it https://github.com/typedclojure/typedclojure/tree/main/example-projects/malli-type-providers#2022-03-2507:14Karol WójcikOh Yeah!#2022-03-2510:09Ben SlessHow does it interface with custom registries and custom schemas?#2022-03-2511:42juhoteperiCool!
What is the tooling like nowadays?
Does it require REPL (so it is one the same classpath as the running app) or could it be integrated into clojure-lsp?#2022-03-2514:37ambrosebs@UK0810AQ2 it's a https://github.com/typedclojure/typedclojure/blob/4818f27a4661262b0d8268a6b00bd0363b02dba0/typed/malli/src/typed/malli/parse_type.clj#L196 right now for custom schemas, a big case. Could probably be a multimethod, do you have an example of a custom schema to help me understand the design space? And custom registries, I'm not sure how far it gets me, but I use m/-deref when I find a :ref. That should be enough?#2022-03-2514:38ambrosebs@U061V0GG2 requires a REPL since it macroexpands code.#2022-03-2514:39ambrosebsMaybe if we create a custom macro rule for every macro in clojure.core we can think of a clojure-lsp version.#2022-03-2514:39Ben SlessA good example of custom schemas would be for time types, such as https://github.com/metosin/malli/pull/545
Similarly for other iterop-y parse-y things, like valid URIs#2022-03-2514:44ambrosebsok, I think we need something like (defmethod malli->Type :local-date [_] 'java.time.LocalTime) .#2022-03-2514:45ambrosebsMaybe if the :class were propagated to the schema itself it would be even simpler.#2022-03-2514:50Ben SlessSo if those schemas had a way to query their backing class it would make it easier to participate in typed.clojure?#2022-03-2514:52ambrosebsin the simplest cases, yes. Is a "simple schema" usually tied to a class?#2022-03-2514:53Ben SlessNo, but I could petition for something like "simple class schema" which could back various boring data classes#2022-03-2514:55Ben SlessAnd maybe add in malli a protocol for accessing those#2022-03-2514:55ambrosebsthat sounds perfect.#2022-03-2515:00ambrosebsCan you think of any custom registry case that might be problematic? I just call m/deref to convert the entire schema eagerly.#2022-03-2515:01Ben SlessRecursive schemas?#2022-03-2515:02ambrosebsah yep, I handle those https://github.com/typedclojure/typedclojure/blob/4818f27a4661262b0d8268a6b00bd0363b02dba0/typed/malli/src/typed/malli/parse_type.clj#L236#2022-03-2515:03Ben SlessShould be fine
How about dependent schemas?#2022-03-2515:03ambrosebsExample?#2022-03-2515:05Ben Slesshttps://github.com/metosin/malli#content-dependent-simple-schema#2022-03-2515:07ambrosebsprobably would need a custom m/type dispatch in the (future) malli->Type multimethod.#2022-03-2515:10ambrosebsanother way these simple schemas could be automated is if Typed Clojure could know the var backing the :pred. eg., pos-int? here#2022-03-2515:11ambrosebsbecause the checker has the annotations for these preds#2022-03-2515:12ambrosebsbut there's a lot of info in the json-schema props too.#2022-03-2515:16ambrosebsperhaps a :type-properties :typedclojure/type convention might be useful too.#2022-03-2517:23ikitommithis is great 🙂#2022-03-2521:41ambrosebsthanks Tommi 🙂#2022-03-2517:19ikitommiwelcome .pretty/prettifier, thanks @meditans for the idea.
Using it to create pretty checker:
(require '[malli.instrument :as mi])
(require '[malli.dev.pretty :as pretty])
(defn check
"check that emits pretty results"
([] (check nil))
([options] ((pretty/prettifier ::check "Check Error" (fn [] (mi/check)) options))))
Dummy pretty printer for ::check:
(defmethod v/-format ::check [_ _ results printer]
{:body (->> (for [[f error] results
:let [{:keys [schema errors]} error
explanation (-> errors first :check :malli.generator/explain-output)]]
[:group
(pretty/-block "When calling:" (v/-visit (-> errors first :check :smallest first (conj f)) printer) printer) :break :break
(pretty/-block "We get:" (v/-visit (:value explanation) printer) printer) :break :break
(pretty/-block "The problem is that:" (v/-visit (me/humanize explanation) printer) printer) :break :break
(pretty/-block "Schema:" (v/-visit schema printer) printer)])
(interpose [:group (v/-color :title-dark (apply str (take 30 (repeat "-"))) printer) :break :break])
(into [:group]))})
in action:
(check)
#2022-03-2517:21ikitommithe actual printer is silly here, should separate the input & output schema problems, looking forward to seeing what you @meditans cooked up for this.#2022-03-2517:23ikitommibut, the core-lib support pretty-anything now. the helper looks like:
(defn prettifier [type title f options]
(let [printer (assoc (or (::printer options) (assoc (-printer) :width 60)) :title title)
actor (::actor options reporter)]
(fn [& args] (when-let [res (apply f args)] ((actor printer) type res) res))))#2022-03-2519:45CarloI am in fact still at a prototype stage re: precise messages (and I mainly produce hiccup to pass to #portal, which I'm trying to turn in my repl for visualization). But I'll definitely try this out and upstream improvements if I end up making them for myself.
I have to say, ideally for me, this prettified error message would be more usable in the form of a data structure (from which I can generate hiccup). But I can as well keep producing it myself from the error!#2022-03-2520:33ikitommiI think a shared function of check-explanation to some intermediate map would be good and could be used by both the malli pretty-printer and your portal visualizer.#2022-03-2615:47CarloI got an error about trying to import malli.instrument in cljs, and in fact I see that in the malli repo we have only a instrument.clj file. Does this mean that I can't do instrumentation in cljs?#2022-03-2620:41ambrosebshave you tried malli.instrument.cljs? It's apparently seen some bugfixes since the last release so I don't know if it works, but I happened upon it a few days ago https://github.com/metosin/malli/blob/master/src/malli/instrument/cljs.clj#2022-03-2620:56CarloThank you, you're right! Here's how I missed that: I looked at this folder https://github.com/metosin/malli/tree/master/src/malli and I could only find the clj version. So I wrongly assumed that was the only available possibility!#2022-03-2620:57CarloHow does the naming schema in the instrument folder work? I have never seen something like this!#2022-03-2702:28ambrosebsI'm guessing that malli.instrument contains functions for the clojure impl. the cljs impl must be implemented via macros (but otherwise with the same names), so a new namespace malli.instrument.cljs was created.#2022-03-2720:19CarloI'm still getting this at times, though:
------ REPL Error while processing ---------------------------------------------
(ns couperin.user
(:require [portal.web :as p]
[malli.core :as m]
[malli.instrument :as mi]
[malli.dev :as dev]))
The required namespace "malli.instrument" is not available, it was required by "couperin/user.cljs".
"malli/instrument.clj" was found on the classpath. Maybe this library only supports CLJ?#2022-03-2720:22CarloSame thing for the namespace#2022-03-2720:25Carloah it's probably a weird interaction with shadow-cljs https://clojureverse.org/t/problem-using-malli-clojurescript-instrumentation-and-shadow-cljs/8612/3#2022-03-2803:42dvingohey, I contributed the cljs support, it's in need of some documentation. When using the instrumentation for cljs the namespaces should be:
[malli.instrument.cljs :as mi]
[malli.dev.cljs :as md]
and there is kondo support but it must be executed at runtime (because the schemas aren't available during compilation (macroexpansion)) and prints to the console, with the intention being you copy it to a kondo config file.
https://github.com/metosin/malli/blob/400dc0c79805028a6d85413086d4d6d627231940/src/malli/clj_kondo.cljc#L203
the js console errors need some work (the pretty one is designed with terminal emulators in mind, not js web console) but if you execute an instrumented fn via nrepl in an editor the error output there is better.#2022-03-2812:02CarloThank you @U051V5LLP, I'm a bit confused on how instrumentation works on cljs. Even after importing:
(:require [portal.web :as p]
[malli.core :as m]
[malli.instrument.cljs :as mi]
[malli.dev.cljs :as dev])
doing
(comment
(dev/start!))
and adding:
(m/=> -add [:=> [:cat :int :int] :int])
(defn -add [x y]
"abcd")
I can still execute calls like:
(-add 1 "a")
getting "abcd" as my answer (and I see that I have the instrumentation message in the browser console:
..instrumented
{ns: 'couperin.user', name: '-add', str: 'couperin.user/-add', _hash: 150786855, _meta: null, …}
just not any error whatsoever#2022-03-2812:28CarloOk, if I call (dev/start!) just before a call to -add, then I get the errors, but if I then modify the definition of -add , the instrumentation is not redone automatically#2022-03-2813:44dvingoinstrumentation works by replacing the function implementations so when hot reload happens the original functions will be replace the instrumented code.
https://shadow-cljs.github.io/docs/UsersGuide.html#_lifecycle_hooks
you'll want to call instrument! or start! after the hot code reload runs (usually something like (defn ^:dev/after-load refresh [] (md/start!)...) for shadow.cljs#2022-03-2814:08CarloThank you, would you still advice to do https://clojureverse.org/t/problem-using-malli-clojurescript-instrumentation-and-shadow-cljs/8612/2 ? I can get it to check something but it's clearly confused by namespaces. Is there a particular place the refresh function you just mentioned should live?#2022-03-2900:22dvingojust the standard setup you'd have for any cljs (utilizing react) app:
https://github.com/day8/re-frame/blob/69cf39552715fa410e7007b7fcbc894097d8db1f/examples/todomvc/src/todomvc/core.cljs#L57#2022-03-2711:20Ivan FedorovNew pretty explainer is so awesome! Great work, thanks a lot!#2022-03-2711:21Ivan Fedorovcan we expect it in cljs land?#2022-03-2813:31Carlothe new pretty explainer seems to break cljs instrumentation, because of not found variables.#2022-03-2816:17rovanionI am either missing something or I've found a bug. It seems like updating a key in a map-spec makes the the optional property have no effect, even though it is still there:
;;; Specification registry
(def registry
(atom {}))
(defn register! [type ?schema]
(swap! registry assoc type ?schema))
;; Combine the default registry with our own mutable registry.
(mreg/set-default-registry!
(mreg/composite-registry
(mreg/fast-registry (malli/default-schemas))
(mreg/mutable-registry registry)))
(register! :db/f
[:map
[:key {:optional true} [:double]]])
(register! :user/f
(-> (malli/deref :db/f)
(mutil/update :key #(mutil/update-properties % assoc :disabled true))))
(malli/deref :db/f)
;; => [:map [:key {:optional true} :double]]
(malli/deref :user/f)
;; => [:map [:key {:optional true} [:double {:disabled true}]]]
(malli/explain :db/f {})
;; => nil
(malli/explain :user/f {})
;; => {:schema :user/f, :value {}, :errors ({:path [0 :key], :in [:key], :schema [:map [:key {:optional true} [:double {:disabled true}]]], :value nil, :type :malli.core/missing-key})}
(mgen/sample :db/f)
;; => ({} {} {} {} {:key -1.5} {} {} {:key -1.09375} {:key -2.0} {:key 4.625})
(mgen/sample :user/f)
;; => ({:key -2.0} {:key -0.5} {:key 2.0} {:key -0.75} {:key -1.5} {:key 1.75} {:key 3.53125} {:key 1.25} {:key -0.25} {:key 0.271484375})
Yup, another user already found it: https://github.com/metosin/malli/issues/645#2022-03-3013:25CarloI'm really liking the pretty explainer namespace, @ikitommi! Do you think this function https://github.com/metosin/malli/blob/d69e06326662c675a40095c7c72ca80af8a8d282/src/malli/dev/pretty.cljc#L69-L75 could get some more inputs? Like, the name of the function that's displaying the error, maybe the line location? Or is it a thing I should try to get via other means?#2022-03-3112:04CarloHere's a tentative PR for this feature @ikitommi, I'd love to know if this is the right direction and what could be improved
https://github.com/metosin/malli/pull/680#2022-04-1310:18ikitommiwill look into this.#2022-03-3102:20mafcocincoIs it possible to supply an arbitrary generator function to an :fn schema (or any schema really)? I know this is trivial, but works for an example:
(malli.g/generate [:map [:a [:fn {:gen/function (fn [] 1)} #(int? %)]]])#2022-04-1310:17ikitommitry :gen/gen.#2022-03-3102:20mafcocincomalli throws an exception with that example, so obviously not correct but was hoping there was something like that.#2022-03-3102:22mafcocincoThe reason I ask is that we have, for example, a customer schema with first-name, last-name, etc. The schema for first-name uses string? to validate but would like to constrain the generated values to a handful of simple names rather than the universe of all possible strings.#2022-03-3102:22mafcocincoWould definitely make the generated data a bit more readable and user friendly.#2022-04-0310:29CarloQuick question: does generative checking work in cljs? If not, why not? If I try to write (mi/check) I get:
------ WARNING #1 - :undeclared-var --------------------------------------------
File: /home/carlo/code/clojure/visualizerTestCljs/src/main/core.cljs:12:1
--------------------------------------------------------------------------------
9 | (defn badd [x y]
10 | "z")
11 |
12 | (mi/check)
-------^------------------------------------------------------------------------
Use of undeclared Var malli.generator/check
--------------------------------------------------------------------------------
edit: solution in thread#2022-04-0310:43CarloOk, it works if I manually include an import on malli.generator , even if I don't directly use the namespace. This is probably due to how shadow-cljs includes the files. I'm going to leave it here in case it's useful to someone else in the future.#2022-04-0400:50dvingoI made an github issue to fix this. you shouldn't have to require that ns - I'll fix it in the coming days#2022-04-0509:18Ferdinand BeyerIs there a recommendation when to call m/schema and when not? I want to define schemas in vars, would wrapping them in in m/schema improve performance?#2022-04-0509:26Ben SlessAlthough I worked on improving parsing performance, calling schema will help with that
It will also ensure the same object is shared if the schema is used multiple times#2022-04-0512:02Ferdinand BeyerOK thanks. I’ve seen that I can get the form with m/form if I need to 😉#2022-04-0718:19respatializedIs there a way to negatively specify a map key? Like "this map cannot contain the :ident key?"#2022-04-0718:20respatializedI could do an :and schema with a :not but that seems clunky and was wondering if there's another way#2022-04-0721:26ambrosebsAFAIK think it's like spec in this respect. There are hacks around it but it's not supported, https://github.com/metosin/malli/blob/3599fbd7fe1eba96c35e5c39073397cba0984b6a/src/malli/core.cljc#L975-L982 is concerning presence, not absence.#2022-04-0721:26ambrosebsanother hack I remember from spec is using map-of's keys with a restricted spec for keys.#2022-04-0721:29ambrosebswhen I https://github.com/typedclojure/typedclojure/blob/bb973bce811c111a814aad4889aa2a083a5e761d/typed/malli/src/typed/malli/parse_type.cljc#L148-L151 a Typed Clojure => malli translation, the best I could find was {:closed true} to prevent all other keys.#2022-04-0721:29ambrosebsThat might be what you want actually.#2022-04-0721:40respatializedUnfortunately not, my use case is a data model like GeoJSON where it's open by default but some keys are reserved and cannot be used in certain contexts#2022-04-0807:43Ben SlessYou can require a map be closed in malli#2022-04-0718:20respatializedI could do an :and schema with a :not but that seems clunky and was wondering if there's another way#2022-04-0807:28Ferdinand BeyerWhat’s the preferred way of testing schema compliance in tests?
Works but does not give helpful failure messages:
(is (m/validate ,,,))
Better, but not super intuitive:
(is (nil? (m/explain ,,,)))
Is there some other way that I’m missing?#2022-04-0807:42Ben SlessInstrument functions and use generators#2022-04-1012:33Martynas MHey.
There is a guy that asks for more transparency on what the library supports. I'm not too sure how to answer his question.
This probably means that he'd expect to either be dismissed and pointed into README once again or then the README should be updated.
https://github.com/metosin/malli/issues/652
https://www.reddit.com/r/Clojure/comments/tykor2/malli_schema_questions/#2022-04-1016:23ambrosebsI took a crack at answering.#2022-04-1015:11pithylessThere is a theory of documentation, that I first saw promoted by Jacob Kaplan-Moss (highly influential in Django documentation and later Heroku Developer Center), that divides documentation into 4 kinds: tutorials, how-to guides, reference and explanation. In general, when I see people struggling with documentation, it is often because the author of the document in question and the reader are in two different quadrants - neither is wrong, but they're missing each other instead of mind-melding.
@invertisment_clojuria - having skimmed the linked issues, I think the same mismatching of expectations is happening. Malli does have some tutorial-like and explanation-like content in the README, and one can explore the codebase and tests to gather more reference-like content. https://malli.io gives examples one may consider how-to guides for modeling common schemas. But I think it is a fair assessment that all these things are scattered and not easy to find, especially for a newcomer to the library.
Aside from Jacob Kaplan-Moss talks and writing you can find online, I found a good write up of the problems with structuring documentation here: https://documentation.divio.com/ I think moving forward, malli would benefit by taking some pointers for how to organize different kinds of documentation and information for the community. It's not a question of what should be in the README, tests, etc., but more of a question of how to organize this information that a reader coming to the project with different mindset and expectations can find the information that is most relevant for them at the time.#2022-04-1208:39eskosHi, did a bit of a hit&run yesterday with this, but wanted to comment 🙂
I totally agree with this! Documentation, in general, is both incredibly hard and incredibly valuable, and I do use that divio link every time I talk about software documentation to anyone as reference - I don’t claim it to be perfect, but it is definitely good enough.
For me, malli’s documentation currently has a discoverability issue. Over the past few years malli has gained a daunting amount of features, but there’s no clear path to learning each feature, and the readme’s examples aren’t comprehensive - this latter is especially kinda amazing considering the readme is probably the longest I’ve ever seen in a Clojure project, especially when considering the amount of examples.
To generalize what I want from documentation is three things:
1. How do I use this. This is always the most critical one, and a sort of blindfold approach - if there isn’t a direct “do x to get y, do z after doing y to get z” kind of How-To guide, I’ll probably never learn about That One Awesome Feature. This is sort of related to the eternal discussion of if left fold is enough or should we also have map and reduce, or is filter enough or do we need keep and remove as well.
2. What is the internal logic. I don’t mean algorithms or stuff like that, but why are things organized in the way they are and what are the architectural decisions. “We like interceptors over middleware” sounds nice, but that assumes I know what interceptors and middleware are. Another would be naming schemes used to name functions, especially prefixes and suffixes used, stuff like that, or something thankfully not that common in Clojure land, but do you return raw values, lazy seqs, promises, channels...how do you assume your library is being used from the outside?
3. All I need is the documentation. If I need to read the source to understand software behavior, the documentation sucks. I’ve been told especially about this that I am Wrong™, but I don’t have time nor interest for that debate - this is my approach 🙂
From technical point of view IMO malli should add one of those libraries which can evaluate code snippets in README just to make sure the snippets stay current and working. Sort of related but inevitable is also the fact that malli’s build is now build+deps based, which is like https://github.com/Gant/Gant all over again, but ehhhh pointing back to my previous points, if I need to worry about how malli is built, I’m already too deep… 🙂#2022-04-1212:04pithylessThanks for the long and insightful comment @U8SFC8HLP :)
It's going to be a slow start, but I'm willing to try to help out with this kind of documentation in the coming weeks. So the core team can focus on delivering more awesome features. ;)
I see Tommi is vacationing atm, but I wonder if other malli contributors or users have more insights on what they'd like to see. I'll start taking some notes.#2022-04-1310:16ikitommi@UK0810AQ2 , you had a tutorial almost done? PR would be most welcome. Also, @U05476190 & all, all documentation improvements are most welcome, /docs being a good place for them.#2022-04-1110:56Martynas MHey.
Is there a way to specify that a schema decoder should decode into a list instead of a vector?
I'd like to have this schema:
[:list :string]
Is this possible without additional code (be it in the transformation step or after the decoding)?
I think a decoder could pick this one up but it would be a custom one:
[:vector {:type :list} :string]
But I simply want to not add any code at all, if possible (maybe to serialize the schema later, not sure).
(type (malli.generator/generate [:sequential :nil])) => clojure.lang.PersistentVector#2022-04-1310:14ikitommicurrently, no. but would be nice. there is a collection-transformer but it’s current purpose is to ensure the right format, should read the :type etc. property for the correct target type#2022-04-1310:00Yehonathan SharvitHello,
We are building an automatic form generator based on Malli.
The form generator input is made of:
• A malli schema
• A structural description of the form
The structure of the form is not necessarily the same as the structure of the data.
Here is a example
{:schema [:map
[:username {:title "Username"} :string]
[:email {:title "Email"
:description "The business email of the user"}
:string]
[:personal [:map
[:age {:title "Age"
:description "The age of the user"} :int]]]]
:ui {:sections [{:title "General"
:fields [{:path [:username]}]}
{:title "Details"
:fields [{:path [:personal :age]}]}]}}
Now, my question is: is there a way to retrieve the schema that corresponds to a map field?
For instance the schema that correspond to the [:personal :age] field.#2022-04-1310:07ikitommisure, but there can be many schemas behind a path, if there is an :and or :or. but if you know what you are doing, you can just the the first one, like this:
(defn schema-in [?schema path]
(let [schema (m/schema ?schema)]
(->> (mu/in->paths schema path) (first) (mu/get-in schema))))
(schema-in
[:map
[:username :string]
[:email :string]
[:personal [:map
[:age :int]]]]
[:personal :age])
; => :int#2022-04-1310:09ikitommibtw, just doing the same thing in a projects (again), would like to push some parts back to the library#2022-04-1311:03Yehonathan SharvitWhat exactly are you doing in your project? UI generation?#2022-04-1312:35ikitommiyes, two cases:
1. technical ui’s for admin/prototyping, directly from malli schemas
2. having large amount of dynamic forms & rules in project(s) => need both utilities for malli-backed form components and some data-oriented generic form-markup, “the ui-schema”
… looking at your example, you are doing 2 too and I know there are many others doing that, looking forward to see if there could be something reusable / shared with these.#2022-04-1316:15Yehonathan SharvitNice!#2022-04-1316:16Yehonathan SharvitHave you found a way to deal with :multi?#2022-04-1408:40Yehonathan Sharvit?#2022-04-1310:07ikitommisure, but there can be many schemas behind a path, if there is an :and or :or. but if you know what you are doing, you can just the the first one, like this:
(defn schema-in [?schema path]
(let [schema (m/schema ?schema)]
(->> (mu/in->paths schema path) (first) (mu/get-in schema))))
(schema-in
[:map
[:username :string]
[:email :string]
[:personal [:map
[:age :int]]]]
[:personal :age])
; => :int#2022-04-1518:53hjrnunesHi. Say I want to validate a sequence of maps and make sure there's at least one of them with a specific entry. Is this in scope for Malli? How would I go about it? Example:
;; my schema is something that somehow looks for [:b 2] in a map sequence
(m/validate my-schema [{:a 1} {:b 2} {:c 3}])
=> true
(m/validate my-schema [{:a 1} {:b 2} {:b 2 :c 4} {:c 3}])
=> true
(m/validate my-schema [{:a 1} {:c 3}])
=> false#2022-04-1519:00Noah Bogartyou can use https://cljdoc.org/d/metosin/malli/0.8.4/doc/readme#fn-schemas to validate "anything":
(def my-schema
[:and
[:sequence map?]
[:fn (fn [s] (some #(= 2 (:b %)) s))]])
i haven't tried that but it should work#2022-04-1520:03Luke JohnsonI’m sure this has been asked before, but is there a way to generate a value compared to the generated value of a key in the same map schema?
{:min-value 250 ;; generated
:max-value 350} ;; always exactly 100 more than min
Looking through the documentation, I can’t find anything specific. https://github.com/metosin/malli/blob/master/docs/tips.md#dependent-string-schemas looks promising or maybe using :gen/fmap but I could really use some guidance.#2022-04-1614:14Ivan FedorovCan I define registry props with options and then reuse these options when I reference props in :map definitions?#2022-04-1817:35mauricio.szaboHi, I was playing with instrumentation on Malli, but I can't find a way to define a function that accepts two args: one int? and other a tuple of int? and string?. Essentially, I want to allow a function to accept (some-function 10 [20 "string"]) as arguments. How do I do this?#2022-04-1818:20ikitommimaybe:
(m/validate [:cat int? [:tuple int? string?]] [10 [20 "string"]])
; => true
#2022-04-1914:23mauricio.szaboYeah, that worked, thanks! I was trying with repeated :cat but that got me some weird messages......#2022-04-2108:28dharriganI'm wondering, after looking at the examples, I still can't really figure out how to decode a string into lowercase. I have something like this:#2022-04-2108:28dharrigan(def foo [:map
{:closed true}
[:name [:string {:max 256 :error/message "should be at most 256 characters"}]]
[:email-address [:string {:decode/string 'clojure.string/lower-case :min 5 :max 100 :error/message "should be between 5 and 100 characters"}]]])
(m/validate foo {:name "foo" :email-address "#2022-04-2108:29dharriganWhat my objective is (in tandem with reitit), is to ensure that the email address coming in is all in lowercase. Bit of a puzzler.#2022-04-2108:29dharriganAny pointers would be appreciated. Thank you.#2022-04-2109:05dharriganSo, got it to work, after plugging away at it for a bit:#2022-04-2109:05dharrigan(m/decode signup-request {:name "foo" :email-address "#2022-04-2109:06dharrigandifference being to use a string-transformer not a json-transformer#2022-04-2109:18dharriganAlthough, it doesn't appear to work with reitit and it's use of json-transformer.#2022-04-2108:51Roee MazorHi, I am looking at this example:
(m/validate [:alt keyword? string?] ["foo"]) ; => true
and I am wondering why it works that way, why not:
(m/validate [:alt keyword? string?] "foo") ; => true
#2022-04-2108:52Roee Mazor(the second one says false which means the alt thing actually makes it look for a sequence for some reason, how can I use :alt without a vec/seq?)#2022-04-2109:01Martynas MTry :or#2022-04-2109:02Roee Mazorperfect, that works perfectly ❤️#2022-04-2113:08Ben SlessAlt is for sequence schemas. It's the | operator in regular expressions#2022-04-2403:28minosniuHi, I'm getting a strange error when running the http://malli.io example of "Pet" in Clojure (not Clojurescript). Code here:#2022-04-2403:29minosniu(def PetMalli
[:schema {:registry {"Pet" [:map
[:type keyword?]
[:name string?]]
"Cat" [:merge
"Pet"
[:map
[:type [:= "Cat"]]
[:huntingSkill [:enum {:description "The measured skill for hunting"}
:clueless, :lazy, :adventurous, :aggressive]]]]
"Dog" [:merge
"Pet"
[:map
[:type [:= "Dog"]]
[:packSize [:int {:min 0,
:default 0
:description "the size of the pack the dog is from"}]]]]}}
[:multi {:dispatch :type} "Cat" "Dog"]])
(m/validate PetMalli {:type "Cat", :name "Viivi", :huntingSkill :adventurous})#2022-04-2403:29minosniuMessage: Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
:malli.core/invalid-schema {:schema :merge}#2022-04-2403:31minosniuCould anyone give me some pointers? We want to use :merge for some inheritance-like constraints in our schema, which is very similar to the "Pet" example.#2022-04-2403:58Ben SlessMerge is in another registry. You need to merge it into the registry you're creating. It's in malli util, I think#2022-04-2404:19minosniuMy guess, too. But the code might get somewhat cumbersome. It's strange that the same code from the Pet example simply won't work, notice that in the example only local registry was used.#2022-04-2404:25Ben SlessYeah that's something I'm still iffy about. Will wait for Tommi to chime in#2022-04-2513:28dvingoI don't think it's that misleading - the code is available here where the initial default registry is setup: https://github.com/metosin/malli.io/blob/master/src/malli/web.cljs#L28#2022-04-2513:58dvingolinking to these sources from the site is probably a good idea though#2022-04-2514:04minosniuWorked like a charm:dizzy:#2022-04-2513:29dvingoI think it's apparent that the local registry being used does not contain all of the schemas available (not just :merge)#2022-04-2514:15ikitommimaybe the util-schema types should be prefixed? e.g. :merge -> :util/merge to make the it more explicit that they are not part of the default registry? btw this works too:
(def Merge (mu/-merge))
(m/validate [Merge [:map [:x :int]] [:map [:y :int]]] {:x 1, :y 2}) ; => true#2022-04-2514:31minosniu:util/merge seems more prominent and alerting.#2022-04-2608:29Martynas MHey.
Is there a way to decode a string into a symbol?
(malli.core/decode [:enum 'identity]
"identity"
decode-transformer)
I want to parse JSON and I want to parse a string into a symbol but I want to force this exact value.
I could define my own decoder and I did it in one other case but I don't think I want to define too many of them.#2022-04-2608:31Martynas MProbably this is the way:
[:and :symbol [:enum 'identity]]
#2022-04-2610:17Ben SlessYou can specify your own decoder in the schema#2022-04-2610:18Ben SlessWhere you can just put the symbol decoder#2022-04-2608:58Martynas MThis fails to generate:
(malli.generator/generate [:and :symbol [:enum 'my-symbol]])
Should I create an issue?#2022-04-2611:31ikitommioh, that’s an test.check impl detail, not sure how much we can do for it.#2022-04-2611:31ikitommie.g. when multiple constraints, first one is used for generation and the rest to filter that the first one generated valid values.#2022-04-2611:32ikitommiquick & dirty fix, override the generator:
(malli.generator/generate [:and {:gen/schema [:enum 'my-symbol]} :symbol [:enum 'my-symbol]])#2022-04-2805:59Martynas MI made this function for myself:
(defn exact-symbol [sym]
[:and {:gen/schema [:enum sym]} :symbol [:enum sym]])
I simply use the generator to validate that my schema is correct. If it generates then it is somewhat correct.
My assumption is that if the generator can pick it up then it's a valid definition (not that the result is valid but the definition).#2022-04-2713:33Ferdinand BeyerI might have found a bug Malli’s clj-kondo type config generation when using :re schemas. Instead of expecting a string matching a regular expression, clj-kondo expects an argument to be of regular expression type. Or did I misunderstand something?#2022-04-2715:17dvingocan you share the clj-kondo configuration for that function?#2022-04-2715:18dvingothe kondo code outputs :regex for :re schemas: https://github.com/metosin/malli/blob/a3db330eae029863e9be443cda3a2cafd8f61a33/src/malli/clj_kondo.cljc#L101 and I'm not certain but I think kondo expects a string: https://github.com/clj-kondo/clj-kondo/blob/d9fca2705863e3e604e004ccb942e0b3d2e268ec/src/clj_kondo/impl/types.clj#L20#2022-04-2715:49Ferdinand BeyerUnfortunately I deleted this example file already, but I can confirm that I get :regex in the config, and I agree that this should probably be :string#2022-04-2715:49Ferdinand BeyerHaving said that, I never worked with clj-kondo that deeply so I don’t know what ops/types it accepts#2022-04-2717:50dvingolooks like the :regex type in kondo expects a regex:
https://github.com/clj-kondo/clj-kondo/blob/master/doc/types.md
added this to the config:
$ cat .clj-kondo/config.edn
{:linters
{:type-mismatch
{:level :warning
:namespaces {foo {foo {:arities {1 {:args [:regex] :ret :string}}}}}}}}#2022-04-2717:50dvingoand then:
$ clj-kondo --lint - <<<'(ns bar (:require [foo :refer [foo]])) (foo "")'
<stdin>:1:45: warning: Expected: regular expression, received: string.
linting took 12ms, errors: 0, warnings: 1
$ clj-kondo --lint - <<<'(ns bar (:require [foo :refer [foo]])) (foo #"")'
linting took 11ms, errors: 0, warnings: 0
#2022-05-0207:35Ferdinand BeyerI created a PR with a tiny fix for this here: https://github.com/metosin/malli/pull/701#2022-05-0208:51ikitommiMerged, thanks!#2022-05-0208:55Ferdinand BeyerThat was fast, thanks 🙂#2022-04-2718:17Keith HouserWhy does humanize not have "Wrong!!" message when wrong value is in the first position?
({:forms (d),
:human [["should be a" "should be b" "should be c"]]}
{:forms (a d c),
:human [nil ["should be a" "should be b" "should be c" "Wrong!!"]]})#2022-05-0207:35Ferdinand BeyerI created a PR with a tiny fix for this here: https://github.com/metosin/malli/pull/701#2022-04-2817:56dumratWhy?
(m/validate
[:map
[:type :keyword]
[:name {:min 1 :max 128} string?]]
{:type :ledger :name ""}) => true#2022-04-2818:01dharriganTry this:#2022-04-2818:01dharrigan(m/validate
[:map
[:type :keyword]
[:name [:string {:min 1 :max 128}]]]
{:type :ledger :name ""}) ;; false#2022-04-2818:01dharrigan(m/validate
[:map
[:type :keyword]
[:name [:string {:min 1 :max 128}]]]
{:type :ledger :name "foo"}) ;; true#2022-04-2908:13rayatNeed help:
(shadow) CLJS + Malli + console multiline printing weirdness - really cannot figure this out.
More details in thread ⬇️
But TLDR: `pretty/reporter` is doing something really crazy, it prints every single character/"word" (or fipp document node, if I'm not mistaken) on new lines creating MASSIVE illegible walls of text in my console. #2022-04-2908:20ikitommiare you using the latest code from master? I believe that is fixed.#2022-04-2908:21rayat> I've also tried other reporters, including just straight console log/warn/error, but since pretty/reporter is the closest to working and the one whose intended output I want, I'll focus on that one.
I’ve tried pretty/reporter, console.log/error/warn, pretty/thrower and have even copied and modified tons of the function call chain of pretty/reporter However:
1. With thrower and default -fail completely illegible schema failures are printed
a. It is extremely "horizontal/squished"
b. and more importantly even inspected in devtools repl, I cannot see the schema failure contents, just the data of the schema I wrote in the first place
i. (I've expanded all the folding bits in devtool's custom formatted content for the error)
2. I also tried replacing some logic within pprint-document (more on that fn to follow) to console.warn instead, and got some progress somehow
a. This is the first screenshot
b. But as you can see, it looks like it's printing actual characters, instead of what I believe might be encodings for whitespace?
3. And with pretty/reporter, it looks like something about the whitespace is messed up, and so is also nearly illegible
a. (second screenshot)
b. The content does print, but it looks like each word boundary whitespace is printed as a new line, and most of the time, it's several empty newlines per.
c. As I step through pprint-document, it looks like after the document goes through (serialize document) and the (eduction chain, the print call in (run! print) receives each individual word, instead of the lines that (I think) should have those words in them
(->> (serialize document)
(eduction
annotate-rights
(annotate-begins options)
(format-nodes options))
(run! print))
a. I thought it might have been the println line after the bit I just posted, but it just (as expected) added an extra newline after all the words from the schema error/document had been newline-printed
b. In the screenshot you'll see console.group being used from one my modifications, just for convenience against scrolling through massive number of lines in the console
I'm really confused and stuck by this, and can't seem to find any mention of this in the malli docs, api browser, GitHub issues and slack search about this, or if anyone else has faced this.
I've tried:
1. explicitly setting the print fn to be console.log/error/warn, even window.alert to see if there was some global pollution of my console var making it split several times per word
2. inspecting the contents of document as passed to format-nodes and pruning bits of the latter's cond conditions/code to only address :group, :text etc
3. opening a fresh new chrome profile with no extensions or anything
4. removing any and all scripts from our <head> like tracing/analytics tools
Maybe it's something about my work stack or setup or something. I'm new to clojure though, and so I've never created any clojure projects of my own, let alone shadowcljs web-apps. If it is required tho, I can try looking into making a repro project
Does #2022-04-2908:21rayat@U055NJ5CC, I have [metosin.malli LATEST] in lein, do you mean a direct git dependency or something?#2022-04-2909:34rayatOh crazy, I only just found the https://github.com/metosin/malli/issues/675.
Unfortunately at my work we use lein, which has no native git-based dependency support. I think I'd have an easier time justifying the introduction of malli itself versus lein plugin into our work codebase, or split our lein config into using tools.deps for dependencies.
Is there
If not, I understand. But then I ask if there's any recommendation you can give to get as much of the latest master's fixes on this bug?
Thanks again.#2022-04-2909:34rayatOh crazy, I only just found the https://github.com/metosin/malli/issues/675.
Unfortunately at my work we use lein, which has no native git-based dependency support. I think I'd have an easier time justifying the introduction of malli itself versus lein plugin into our work codebase, or split our lein config into using tools.deps for dependencies.
Is there
If not, I understand. But then I ask if there's any recommendation you can give to get as much of the latest master's fixes on this bug?
Thanks again.#2022-04-2914:27dvingoyep very recent fix: https://github.com/metosin/malli/commit/d1a28f734e6fa3e2f669d5cb08ecfcd0558991de#2022-04-2914:58dvingobeing stuck on lein is unfortunate, I led a project off of lein to deps across multiple repos last year and know that pain.
however, thanks to Tommi's excellent design of having the reporter be a function you provide, you can address this in your own codebase:
https://github.com/metosin/malli/blob/a3db330eae029863e9be443cda3a2cafd8f61a33/src/malli/dev/cljs.cljc#L32
just copy this ns:
https://github.com/metosin/malli/blob/master/src/malli/dev/pretty.cljc
and this one:
src/malli/dev/virhe.cljc
to include the relevant fix:
https://github.com/metosin/malli/commit/d1a28f734e6fa3e2f669d5cb08ecfcd0558991de#diff-78b0ed5a31a32de704a1f77385e434d91bc5a44a0f74d37bd313ea96a130d97cR27
it should be noted there were two even more recent fixes for cljs though. the implication is that things will work without those fixes only for :malli/schema metadata style function schemas
but that should get you unblocked!#2022-04-2916:16rayatThanks so so much for the help! I've been trying to figure this out for like a week, too embarrassed in case it was like an extension or external script or some user error that was causing it - I didn't even think to check if the latest master/docs were released or not.
Regarding your points:
1. facepalm I was trying to copy individual functions lol, can't believe I didn't think to just copy the relevant ns completely. Will do this post-haste.
2. Of those 2 recent fixes for cljs, it looks like
a. https://github.com/metosin/malli/commit/f7b4e12d3fe3e2f60ff50057118308b7a3abe148 is about collecting metadata fn schemas, doesn't that mean it's a fix only for metadata style fn schemas? So wouldn't things work without this fix if I didn't use metadata style? ( the inverse of what you said)
Lastly, since I'm here:
1. I'm actually using neither style, as I'm trying to use malli with helix defnc components.
a. I'm directly using a wrapper over (-instrument {:schema ... :reporter ...} some-fn-arg-from-wrapper)
i. Would that have some implications regarding any of the since-last-release cljs fixes?
b. Without too many specifics, but in case helps to know why,
i. defnc does not forward metadata to the expanded fn, so that route doesn't work, would have otherwise used that
1. (yet again, there's an unpublished recent change that added metadata forwarding lol - oh lein)
ii. the output of the defnc macro could sometimes be an Fn , in which case => would work, but at other times it produces a js obj with a key render that has the equivalent Fn
1. This is just a detail of React, wherein the latter case of the obj.render is a special case handled automatically by React
iii. so my wrapper just instruments the output of defnc with the provided schema for this special case, so it picks out the render key's val if it's an obj, or the fn itself if it's an fn, and then passes either to -instrument #2022-04-2917:27dvingofor 2. even though it's about collect it doesn't impact functionality - just developer experience (requires saving your code twice to see new fn schemas)#2022-04-2917:28dvingook cool, I think using -instrument directly should work - for getting around helix not forwarding meta you can structure your code such that the body of all helix components proxy to your own functions of props -> react element#2022-04-2917:28dvingoand then just instrument those#2022-05-0313:22Nikolas PafitisHi, I'm using mr/set-default-registry! and I'm getting the following error when running kaocha test runner and the namespaces reload on change.
Caused by: java.lang.IllegalArgumentException: No implementation of method: :-schema of protocol: #'malli.registry/Registry found for class: nil
at clojure.core$_cache_protocol_fn.invokeStatic (core_deftype.clj:584)
...
malli.registry$eval15830$fn__15831$G__15819__15838.invoke (registry.cljc:11)
malli.registry$custom_default_registry$reify__15870._schema (registry.cljc:50)
malli.core$_lookup.invokeStatic (core.cljc:260)
malli.core$_lookup.invoke (core.cljc:258)
malli.core$_lookup_BANG_.invokeStatic (core.cljc:265)
malli.core$_lookup_BANG_.invoke (core.cljc:263)
malli.core$schema.invokeStatic (core.cljc:1985)
malli.core$schema.invoke (core.cljc:1963)
malli.core$properties.invokeStatic (core.cljc:1999)
malli.core$properties.invoke (core.cljc:1994)
malli.core$properties.invokeStatic (core.cljc:1997)
malli.core$properties.invoke (core.cljc:1994)#2022-05-0508:15pinkfrogIs there any example that uses malli to validate html forms on the frontend side?#2022-05-0515:03Alexis SchadHi there, is there a minimal sample project using Malli instrumentation (during dev) in a shadow-cljs webapp? Or is there a recommended configuration for https://malli.dev in cljs? I'm struggling a bit to setup it well.
-> thread for more info#2022-05-0515:03Alexis SchadWith a basic configuration I got:#2022-05-0515:03Alexis Schad#2022-05-0515:03Alexis Schadwith thrower i got:#2022-05-0515:03Alexis Schad#2022-05-0515:08dvingothe implementation of that is under active development - if you can use the latest git commit you'll have better errors#2022-05-0515:09dvingothis thread has some background, I suggest adding this metadata to your entry ns:
https://github.com/metosin/malli/issues/695#issuecomment-1116884390#2022-05-0515:12Alexis Schad> you'll have better errors
The error seems pretty good rn, it's just a problem of formatting them inside the console/devtools
I'm looking for your link to see if it can improve something#2022-05-0515:21Alexis SchadWith the master version of pretty/reporter, there's a reader conditional for cljs and it now prints well, thanks! I'll see if I have hot reload issue like your link#2022-05-0515:30dvingonice! yea it's tricky because instrument outputs code where it is called that overwrites the fn - so if you're iterating on the implementation of the the fn in a separate namespace they can get out of sync#2022-05-0515:43Alexis SchadI was using dev/start! but I had to use instrument! instead. I think I don't need the {:dev/always true} metadata, it seems to just work fine without it (I call instrument! every time the app is hot reloaded).#2022-05-0515:45dvingoYou'll run into staleness issues described in the ticket without reloading the ns where instrument is called#2022-05-0515:46dvingoif you macroexpand the intstrument call it can help understand why that happens#2022-05-0516:07Alexis SchadI tried to update both the function and the spec and it seems to work well#2022-05-0518:15dvingonice! well alright then lol#2022-05-0520:09Alexis SchadI didn't even thank you… so thanks!#2022-05-0600:02dvingofor sure! no prob - thanks for trying it out! the leverage from malli is pretty remarkable, just helping to add to that power 🙂#2022-05-0700:36Alexis SchadI think I've just had some issues that has been solved with {:dev/always true}
Reloading the webapp was doing nothing, I had to restart my shadow-cljs watch. And with the workaround above it seems to work.#2022-05-0619:38mauricio.szaboHey, @danvingo, I though maybe if you want, we could use a thread here to discuss this PR? https://github.com/metosin/malli/pull/702. I added some comments on how to make the metadata work with Shadow-CLJS, but it only works on ClojureScript :thinking_face:#2022-05-0622:06dvingogot something working!
these macros are fun 😄
thanks for investigating, I thought adding metadata on the symbol would work but I'm guessing the cljs compiler handles functions differently#2022-05-1312:39eskosI’d need a bit of help/ideas to implement conditional validation for a case when
• vector with a single Thing validates as true as it normally would
• vector with multiple Things validates true only if each Thing has an additional field
• Thing is a record represented as map
Practically this is “If only one record is present in result set, id field isn’t required, otherwise id field is mandatory”. There’s probably multiple ways to tackle this, I’m just not sure what would be a good approach.#2022-05-1313:22eskosBasically what I have is
(m/schema [:or
[:vector {:min 1} (mu/merge IdField Thing)]
[:vector {:max 1} Thing]])
and this seems to work as intended but I’m not sure if this is the best way to do this.
EDIT: Ordering also matters, the more complex schema must be first for sensible error output.#2022-05-1613:55ikitommiWould :multi here?
[:multi {:dispatch count}
[1 [:vector {:max 1} Thing]]
[:malli.core/default [:vector {:min 2} (mu/merge IdField Thing)]]]#2022-05-1614:04eskosOh yes, :multi seems to work wonderfully! 👍#2022-05-1613:55ikitommiWould :multi here?
[:multi {:dispatch count}
[1 [:vector {:max 1} Thing]]
[:malli.core/default [:vector {:min 2} (mu/merge IdField Thing)]]]#2022-05-1405:25PrashantI was trying to write a function schema such that specs of second argument are dependent on the value of first argument e.g.
first argument can be a :keyword with value :id or :contact-id
second argument has to be a :map that needs to contain the key supplied in first argument
So far, I have done below:
[:=>
[:cat
[:alt [:= :id] [:= :contact-id]]
[:alt [:map [:id :string]] [:map [:contact-id :string]]]]
:string]
(defn test-fn
{:malli/schema =>test-fn}
[key value]
(str (name key) "_" (key value)))
However, above allows (test-fn :contact-id {:id "op"}) as a valid case in instrumentation as there is no correlation b/w first and second argument.
I would greatly appreciate some help.
EDIT:
Below schema works:
(def =>take2
[:=>
[:cat
[:alt
[:cat [:= :id] [:map [:id :string]]]
[:cat [:= :contact-id] [:map [:contact-id :string]]]]]
:string])
(defn test-fn-2
{:malli/schema =>take2}
[key value]
(str (name key) "_" (key value)))
(test-fn :contact-id {:id "op"}) is invalid with instrumentation on. It is a little verbose though.#2022-05-1611:24vemvI might need a refresher. Given data expressed as strings (because the data comes from something that inherently only expresses things as strings e.g. a spreadsheet or csv) and a richer schema (i.e. attribute x is an int, not a string), how do I coerce those string values into a schema?
Example:
(my/coerce [:sequential [:map [:a int?]]]
[{:a "1"}]) ;; => Produces `[{:a 1}]`, i.e. coerces the string into an int
;; ...and throws if no coercion was possible#2022-05-1611:26ikitommitry decoding: https://github.com/metosin/malli/blob/master/README.md#value-transformation#2022-05-1611:28ikitommiit does not throw by default, but you can validate the result and throw if it's invalid. Two walks on the schema with malli are still much faster than one (decode or throw) with most others.#2022-05-1611:28vemvthanks much! Let's give it a go :)#2022-05-1613:50chen florescu(def example-schema
[:map {:closed true}
[:product {:optional true} Product]
[:version {:default "1.0"} [:= "1.0"]]
[:enable true?]
[:window [:int {:min 12 :max 1512}]]])
I have this schema and I want to add a constraint that only if I receive product (which is optional), then window needs to be [:int {:min 12 :max 24}]. Any suggestions how can I do that?#2022-05-1615:34Ben SlessThose are two different schemas and there's an or between them. Then product won't be option, either#2022-05-1709:03eskosCould a new version be released to clojars? I just spent an embarrassing amount of time wondering why (mt/default-value-transformer {::mt/add-optional-keys true}) isn’t working only to realise it was added 18 days after 0.8.4 was released 😛#2022-05-1812:59Ferdinand BeyerIs there a schema for any homogenous collection, similar to spec’s coll-of? I found :vector , :sequential and :set, and fell into the trap that a set is not sequential?.
I want to spec a function that will take any collection and passes it to (set).#2022-05-1813:05Ferdinand BeyerWould it be worth adding a :coll schema to base-schemas like this?
:coll (-collection-schema {:type :coll, :pred coll?})#2022-05-1911:21armedHey, everyone. What’s the best way to describe homogeneous map but add specific keys in it?#2022-05-1911:22armedI tried :union and :and with no luck.#2022-05-1911:23armed(mu/union
[:map
[:state keyword?]
[:failure {:optional true} any?]
[:value any?]]
[:map-of keyword? string?])#2022-05-1911:29armedWell, actually it’s working. Had to replace :union with mu/union 🙂#2022-05-1913:01ikitommigood to hear it works! I think union puts :or around the two here.#2022-05-1913:02ikitommi:thinking_face:#2022-05-1913:02ikitommibut, there is an issue to solve this elegantly.#2022-05-1922:27aaron51Seems like pretty/explain doesn’t support custom registries?#2022-05-2014:16plinshello everyone, I’m using reitit and malli to validate JSON requests to endpoints of my API
I would like to use the malli registry, in a spec fashion, to define keys first, and afterwards referencing them in a map. doing this way would allow me to https://clojure.org/about/spec#_decomplect_mapskeysvalues
the problem Im facing is: the json payload will spawn un-qualified keys, and the registry only works with qualified keys
what would be the best solution to this problem? I would like to avoid [:key :ns/key]#2022-05-2306:07jprudentHello, I've a question about the behaviour of :and. Here is a schema with a valid input:
(m/explain [:and
[:sequential any?]
[:multi {:dispatch 'first}
[:a [:sequential keyword?]]
[::m/default any?]]]
[:a :b])
=> nil#2022-05-2306:09jprudentbut with invalid input
(m/explain [:and
[:sequential any?]
[:multi {:dispatch 'first}
[:a [:sequential keyword?]]
[::m/default any?]]]
1)
Execution error (IllegalArgumentException) at malli.core/-multi-schema$reify$reify$fn (core.cljc:1497).
Don't know how to create ISeq from: java.lang.Long
I was expecting that the branch that checks that the input is sequential would prevent execution of the second one#2022-05-2306:13jprudentThe solution I found to avoid an exception to be raised is to have smarter dispatch function
(m/explain [:multi {:dispatch (fn [x] (if (seq? x) (first x) ::error))}
[:a [:sequential keyword?]]
[::error [:sequential any?]]
[::m/default any?]]
1)
=>
{:schema ...,
:value 1,
:errors ({:path [:muguet.meta-schemas/error],
:in [],
:schema [:sequential any?],
:value 1,
:type :malli.core/invalid-type})}#2022-05-2306:23jprudentWhy checking branches in an :and is not lazy (could stop after one of the branch is invalid) ?#2022-05-2306:24jprudent(I can live with current behaviour, no problem 🙂 just curious) Thanks#2022-05-2310:06Alexis SchadHi. I'm very not expert in malli, but maybe it's the use of explain: explain try to give all the reasons why it's not valid. Let's say you have a password that must have a specific length, some special characters, etc. You may want to display all the reasons at once instead of a "one by one" strategy. But a lazy version would be useful I think. (I didn't check but validators should already be lazy)#2022-05-2312:30jprudentYou're right, validate is "lazy"!#2022-05-2312:31jprudentYour explanation makes sense, thanks.#2022-05-2311:48rovanionIs it possible to remove a key-value-pair from a map during decode if the value is a specific thing?
I've got a select with an entry that reads "all" which is functionally the same as not giving any value at all and I want to handle that case in the reitit/malli request param coercion stage.#2022-05-2311:52rovanionI think I've found a relevant example:
(m/decode
[string? {:decode/string {:enter 'str/upper-case}}]
"kerran" mt/string-transformer)#2022-05-2311:52rovanion; => "KERRAN"
#2022-05-2311:56rovanionThough that won't remove the key:
(malli.core/decode [:map [:a [:string {:decode/string #(if (= "all" %) nil %)}]]]
{:a "all"}
malli.transform/string-transformer)
;; => {:a nil}#2022-05-2312:44rovanionPara on IRC gave me a solution:
(defn- remove-alla-odlingsplatser
"Takes a query-params map and removes the key :odlingsplatser if its value is 'alla odlingsplatser'."
[m]
(if (= "alla odlingsplatser" (:odlingsplats m))
(dissoc m :odlingsplats)
m)) #2022-05-2312:44rovanionWelll, the solution he gave me was: (fn [m] (if (= "all" (m :a)) (dissoc m :a) m))#2022-05-2315:13William RobinsonI want to use malli/decode on a map that has to contain at least one of :foo/id URI or :bar/id UUID. No matter how I twist and turn the schema, the only thing I can get working is the following:
(malli/decode [:or
[:map
[:foo/id uri?]
other-child
other-other-child]]
[:map
[:bar/id uuid?]
other-child
other-other-child]]]
{:bar/id #uuid "15a0bc27-5725-41d9-89c9-6f8d3966447a"
:other-key :other-value}
(mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
Is there a good way to handle this without having to duplicate the schema?#2022-05-2315:57jprudent[:map
[:foo/id [:or [uri?] [uuid?]]]
[:attr1 :int]
[:attr2 :int]]#2022-05-2315:57jprudentcould that work ?#2022-05-2316:28William RobinsonThanks for the suggestion! Unfortunately they are different keys, one of which always corresponds to a URI and the other to a UUID.#2022-05-2316:32William Robinson[:map
[:or [:foo/id uri?]]
[:bar/id uuid?]]
[:attr1 :int]
[:attr2 :int]]
Something like the above is what I would like to achieve.#2022-05-2317:05jprudent[:union
[:or
[:map [:foo/id uri?]]
[:map [:bar/id uuid?]]]
[:map
[:attr1 :int]
[:attr2 :int]]] #2022-05-2317:07William RobinsonThis looks promising, will try it out and let you know, thanks!#2022-05-2317:08jprudent[:and [:map
[:foo/id {:optional true} uri?]
[:bar/id {:optional true} uuid?]
[:attr1 :int]
[:attr2 :int]]
[:fn (fn [m] (or (:foo/id m) (:bar/id m)))]] #2022-05-2408:49William RobinsonForgot to get back to you, but the last solution worked like a charm, huge thanks!#2022-05-2406:40Ben SlessI'm trying to have mutual recursion with regex schemas and failing pitifully. Any tips?#2022-05-2406:47Ben SlessIn this example:
(defn char-range
[from+to]
(let [from (long (first from+to))
to (long (second from+to))]
(m/-simple-schema
{:type ::char-range
:pred (fn [x] (<= from (long x) to))})))
(defn char-seq
[cs]
(into [:cat] (mapv (fn [c] [:= c]) cs)))
(m/parse
(m/schema
[:schema
{:registry
{::digit (char-range "09")
::lower-case (char-range "az")
::upper-case (char-range "AZ")
::letter [:alt ::lower-case ::upper-case]
::number [:+ ::digit]
::var [:cat ::upper-case [:* ::lower-case]]
::boolean [:altn [:true (char-seq "true")] [:false (char-seq "false")]]
::atom [:cat ::lower-case [:* ::letter]]
::term [:altn
[:equality [:schema [:ref ::equality]]]
[:atom ::atom]
[:var ::var]
[:number ::number]
[:boolean ::boolean]
#_[:structure ::structure]
#_[:list ::list]
#_[:string ::string]]
::equality [:cat ::var (char-seq " = ") ::term]
}}
::equality])
(seq "X = Y"))
::equality succeeds but ::term doesn't#2022-05-2407:31jprudentI'm not sure why but I also find out that you can't have :ref in seqexp. Is that your problem ?#2022-05-2407:33jprudentIs it going in a stackoverflow if you remove it ?#2022-05-2407:34jprudentmaybe try :orn instead of :altn#2022-05-2409:02Ben SlessThat doesn't work#2022-05-2409:04Ben Slessfor some reason you also can't wrap the ::term in ::equality with [:schema [:ref ,,]]#2022-05-2418:31Alexis SchadWhat doesn't succeed? What do you expect?#2022-05-2418:31Alexis SchadI got [[\X []] [\space \= \space] [:var [\Y []]]]#2022-05-2419:14Ben SlessTry replacing ::equality schema with ::term#2022-05-2420:07Alexis Schadho ok#2022-05-2420:08Alexis SchadI think it's because in ::term you altn only the \space and not the whole input seq. It tries to match \space and don't find any entry. I'll try to add an entry to check that#2022-05-2420:18Alexis Schad(m/parse
(m/schema
[:schema
{:registry
{::digit (char-range "09")
::lower-case (char-range "az")
::upper-case (char-range "AZ")
::letter [:alt ::lower-case ::upper-case]
::number [:+ ::digit]
::var [:cat ::upper-case [:* ::lower-case]]
::boolean [:altn [:true (char-seq "true")] [:false (char-seq "false")]]
::atom [:cat ::lower-case [:* ::letter]]
::term [:* [:altn
[:equality [:schema [:ref ::equality]]]
[:atom ::atom]
[:var ::var]
[:number ::number]
[:boolean ::boolean]
[:space [:cat [:= \space]]]
[:equals [:cat [:= \=]]]
#_[:structure ::structure]
#_[:list ::list]
#_[:string ::string]]]
::equality [:cat ::var (char-seq " = ") ::term]}}
::term])
(seq "X = Y"))
This "works" => [[:var [\X []]] [:space [\space]] [:equals [\=]] [:space [\space]] [:var [\Y []]]]#2022-05-2502:45Ben SlessWell, you changed the semantics of the parser. An "equals" is defined in the bnf as "atom = term"#2022-05-2502:45Ben SlessI don't want to do extra parsing afterwards#2022-05-2502:46Ben Sless”X = Y” should be tagged as an equality term#2022-05-2507:18Alexis SchadI know, that's why I put " around works. I was trying to explain the reason why it doesn't work.
Btw you can’t do recursive seqexp in malli actually. If you want to exclusively use Malli to parse, you can replace the ref to ::term with a [:* any?] and call the parser again on that.
You can also create your own schema but it’s harder I think.#2022-05-2508:39Ben SlessWhy can't I have recursive seqex schemas, though?#2022-05-2508:44Alexis SchadDon't know, not a malli expert. But I think it's due to technical limitation. Malli explicitely prevent it with ::potentially-recursive-seqex error, though it is theorically possible.#2022-05-2508:51Alexis SchadSee https://github.com/metosin/malli/blob/85a948afc13262adb2082ede36ee131bd2b6364d/test/malli/core_test.cljc#L1797-L1800
[:schema {:registry {::ints [:cat int? [:ref ::ints]] is disallowed.#2022-05-2705:28escherizeThere’s more info here in this discussion: https://github.com/metosin/malli/pull/317#2022-05-2516:47kennyHi. Instead of an :error/fn returning a string, could I return explain data? If yes, what's the recipe to do so?#2022-05-2516:58Alexis SchadWhat do you mean by "explain data"? But yes if you returns a non-string value, it works as you expect (I think).#2022-05-2612:45plinshttps://clojuredocs.org/clojure.spec.alpha/explain-data#2022-05-2915:14ikitommiCurrently no, but in one project needed richer explanations and did an ugly hack to enable that. I think we should support that at library level. E.g. one could return {:message "so wrong", :level "warning"}#2022-05-2915:15ikitommiCould you write an issue of this?#2022-05-2911:09ingesolIs it possible to make the code below work on the JVM, or is there some limitation in the implementation of CLJS function schemas that only makes them accessible in the JS runtime? My function schemas work and reload nicely in the browser, but in the code below the collect-cljs call always returns () .
Additional info: All my function schemas are :malli/schema metadata.
(defn -main
[& _]
(-> (malli.clj-kondo/collect-cljs)
(malli.clj-kondo/linter-config)
(malli.clj-kondo/save!)))#2022-05-2915:12ikitommiGood question. I have no idea! I believe @danvingo knows.#2022-05-2921:21dvingoif you have a symbol in your functions schema like:
(defn minus
{:malli/schema [:=> [:cat :int] [small-int]] }
[x] (dec x))
small-int can only be resolved at runtime by a js vm.
that's the fundamental limitation to why it won't work on a java vm#2022-05-3004:57ingesolThanks for clarifying, @danvingo. Also, big thanks for all your recent work on the CLJS tooling, it’s made a huge difference!#2022-05-3015:31dvingono problem. Ah that's great to hear! glad it's working for you - interested to hear if you have any feedback or things that could be improved, but if all is well that's great too 🙂#2022-05-3015:57dvingoin case you didn't know there's https://github.com/metosin/malli/blob/384e31b325928a29f6bfdddae3a9f1241be4734b/src/malli/clj_kondo.cljc#L203 which is intended for use in a cljs repl - it's manual, you have to copy and paste to the kondo config.
Are you trying to output the kondo config using a script (somewhat automated)?
One thought I had is that for scripting support we could add a write function for node.js, the only annoying thing is that any browser APIs not available in node.js would probably break things, which would mean you'd have to change the structure of your code to put the function with schemas in one place.
There's tradeoffs in all these approaches unfortunately#2022-05-3016:16ingesolYeah, that sounds a bit too much, I was thinking about node as well but probably not the route to go. My use case: I would like to keep the clj-kondo config in version control. The ideal would be a process that recompiles the config on every code change, so it would be obvious for the developer that the config is changed and needs to be pushed.#2022-05-3016:18ingesolbeing able to print it to console and then copy into the file is not too bad, will probably be fine.#2022-05-3016:20ingesolAs for the general usability of CLJS: I’m using shadow-cljs. I followed the advice in the docs, and things seem to mostly work. I think I stumbled upon a file here and there that wasn’t being updated when schema changed, but it usually is fixed by a refresh.#2022-05-3016:21ingesolDon’t have much detail on that here, as I’m not actively working on that code right now. But I will in a few weeks, will try to investigate properly then. But current state of things is perfectly fine for me!#2022-05-3113:29dvingoI see, the use-case makes a lot of sense, I'm wondering if there is a way to achieve it as part of the build process in CI. The tricky part is you want the code to be evaluated in a browser, but you also want access to the filesystem. I was thinking about something like karma tests - running in a headless browser in node... Not sure if that would work.#2022-06-0115:13dvingoAnother idea to get this a bit more automated:
add a dev-time only button to your app that sends the cljs kondo config to a specific endpoint which writes to the kondo config file - so it's just a button press#2022-06-0115:14ingesolYeah, that could work. Might be our preferred solution in the end 🙂#2022-06-0115:19dvingooh - or just do it in your reload hook! then no interaction needed#2022-06-0205:51ingesolTrue, that’s what I’m doing right now, but of course just printing to console.#2022-06-0205:53ingesolQuestion: I’m exclusively using keywords in schemas, never symbols. At least for now. Would I be able to “cheat” by telling the malli->cljs-kondo API that my cljs codebase is clj?#2022-06-0215:08dvingoyou could put them in cljc files and they'd be picked up when you collect clj schemas. Other than that, take a look at the source, the code is pretty tiny that collects the schemas, so you should be able to get it working (I think) with your own kondo wrapper. The library can't make those assumptions about what user schemas will have in them, but with those assumptions I don't think there's a reason it can't work#2022-05-3004:57ingesolThanks for clarifying, @danvingo. Also, big thanks for all your recent work on the CLJS tooling, it’s made a huge difference!#2022-05-3004:52AbhinavWhen malli transforms a data structure, does it not preserve meta data? is this by design?
(require '[malli.core :as mc]
'[malli.transform :as mt])
(def ^{:a 10} alist (list 1 2 3))
(meta (mc/decode [:vector int?] alist (mt/transformer (mt/json-transformer) (mt/string-transformer))) )
;=> nil#2022-05-3111:58eskosThis might be a bit broad question, but tl;dr is How do I implement a custom transformer?
I think I want to implement a completely custom transformer to take over the entire decode/encode process - that is, I want to entirely bypass what the built-ins do, and bring my own logic to do the value transformation.
I have something brewing elsewhere that I want to integrate to malli which does value type transformation on its own, and for it to work I essentially need a function callback on malli side which would get the relevant malli schema node and its properties (eg. ) + the input value from data to be able to do that transformation.
To me this looks like a job for custom transformer which would be created with the malli.transform/transformer helper function, but I can’t make heads or tails about what kind of code I need to write to actually enable such overriding capture of the entire transformation.#2022-05-3112:20ikitommitry :default-decoder with :compile if you want to prepare the transformer to be fast#2022-05-3112:24ikitommiHere’s a sample, if that’s what you are looking for:
(def transformer
(mt/transformer
{:name :my-transformer
:default-decoder {:compile (fn [schema _]
(println "<<<" (pr-str schema))
(fn [x]
(println ">>>" (pr-str schema) "->" (pr-str x))
x))}}))
(def decode
(m/decoder
[:map
[:x [:set [:enum "S" "L"]]]
[:y :uuid]
[:z [:tuple :boolean [:map [:a :int]]]]]
transformer))
;<<< [:map [:x [:set [:enum "S" "L"]]] [:y :uuid] [:z [:tuple :boolean [:map [:a :int]]]]]
;<<< [:malli.core/val [:set [:enum "S" "L"]]]
;<<< [:set [:enum "S" "L"]]
;<<< [:enum "S" "L"]
;<<< [:malli.core/val :uuid]
;<<< :uuid
;<<< [:malli.core/val [:tuple :boolean [:map [:a :int]]]]
;<<< [:tuple :boolean [:map [:a :int]]]
;<<< :boolean
;<<< [:map [:a :int]]
;<<< [:malli.core/val :int]
;<<< :int
(decode
{:x #{"S" "L" "XL"}
:y :invalid
:z [true {:a 123}]})
;>>> [:map [:x [:set [:enum "S" "L"]]] [:y :uuid] [:z [:tuple :boolean [:map [:a :int]]]]] -> {:x #{"L" "S" "XL"}, :y :invalid, :z [true {:a 123}]}
;>>> [:malli.core/val [:set [:enum "S" "L"]]] -> #{"L" "S" "XL"}
;>>> [:set [:enum "S" "L"]] -> #{"L" "S" "XL"}
;>>> [:enum "S" "L"] -> "L"
;>>> [:enum "S" "L"] -> "S"
;>>> [:enum "S" "L"] -> "XL"
;>>> [:malli.core/val :uuid] -> :invalid
;>>> :uuid -> :invalid
;>>> [:malli.core/val [:tuple :boolean [:map [:a :int]]]] -> [true {:a 123}]
;>>> [:tuple :boolean [:map [:a :int]]] -> [true {:a 123}]
;>>> :boolean -> true
;>>> [:map [:a :int]] -> {:a 123}
;>>> [:malli.core/val :int] -> 123
;>>> :int -> 123#2022-05-3112:31ikitommiIf you want to run bottom-up, use :leave:
(def transformer
(mt/transformer
{:name :my-transformer
:default-decoder {:compile (fn [schema _]
(println "<<<" (pr-str schema))
{:leave (fn [x]
(println ">>>" (pr-str schema) "->" (pr-str x))
x)})}}))#2022-05-3112:31ikitommi=>#2022-05-3112:31ikitommi(decode
{:x #{"S" "L" "XL"}
:y :invalid
:z [true {:a 123}]})
;>>> [:enum "S" "L"] -> "L"
;>>> [:enum "S" "L"] -> "S"
;>>> [:enum "S" "L"] -> "XL"
;>>> [:set [:enum "S" "L"]] -> #{"L" "S" "XL"}
;>>> [:malli.core/val [:set [:enum "S" "L"]]] -> #{"L" "S" "XL"}
;>>> :uuid -> :invalid
;>>> [:malli.core/val :uuid] -> :invalid
;>>> :boolean -> true
;>>> :int -> 123
;>>> [:malli.core/val :int] -> 123
;>>> [:map [:a :int]] -> {:a 123}
;>>> [:tuple :boolean [:map [:a :int]]] -> [true {:a 123}]
;>>> [:malli.core/val [:tuple :boolean [:map [:a :int]]]] -> [true {:a 123}]
;>>> [:map [:x [:set [:enum "S" "L"]]] [:y :uuid] [:z [:tuple :boolean [:map [:a :int]]]]] -> {:x #{"L" "S" "XL"}, :y :invalid, :z [true {:a 123}]}#2022-05-3112:50eskosAlright, thank you! What’s the second argument in :compile (fn [schema _] ... ?#2022-05-3112:56ikitommi_options, passed in all callbacks. Escape hatch to override anything at any time 😉#2022-05-3112:57eskos*danger-zone* 😄#2022-06-0517:08eskosContinuing on this, I’d need a bit of clarification, see snippet:
(malli.core/decode
[:map [:a {:foo :bar} :int]]
{:a "string"}
(malli.transform/transformer
{:name :hello
:default-decoder
{:compile (fn [schema _]
(println (str "node schema: " (malli.core/-form schema)))
(fn [x]
(println (str "value:" x " / schema at val: " (malli.core/-form schema)))
x))}}))
node schema: [:map [:a {:foo :bar} :int]]
node schema: [:malli.core/val {:foo :bar} :int]
node schema: :int
value:{:a "string"} / schema at val: [:map [:a {:foo :bar} :int]]
value:string / schema at val: [:malli.core/val {:foo :bar} :int]
value:string / schema at val: :int
=> {:a "string"}
It seems the :a tuple is walked twice, once for the tuple itself and once for the tuple’s value. I find this confusing, as the associative nature of [:a :int] would imply the properties of the key apply to the value as well.
There’s a pretty easy way to get around this, schema
[:map [:a [:int {:foo :bar}]]]
does provide properties for the :int which really was my intention, and it sort of makes sense, but it’s of course a bit kludgier.
My assumption here was that if, for example, the key is marked optional then its value is naturally optional as well, and as such, all other properties should transfer to values directly as well, plus the value shouldn’t be walked on its own.
So, bug, feature, gotcha or something I could trick the transformer to handle through the aforementioned _options so that it doesn’t walk to the tuple’s value? 🙂
EDIT: I created an issue about this, lets continue on GH#2022-06-0612:27Tiago Dall'OcaHello hello!
Is there a generics implemented in malli in some way? I was thinking maybe e.g. [:generic=> [:n-type number?] [:cat :n-type] [:vector :n-type]], :generic=> taking a vector of pairs of keywords and schemas and then the usual :=> args#2022-06-0613:50Noah BogartAll clojure functions are generic, lol#2022-06-0615:05Tiago Dall'Ocathe idea is to parse malli schemas to ts type definitions#2022-06-0615:05Tiago Dall'Ocato make cljs libraries nicer to consume from ts#2022-06-0615:12Noah BogartSorry, I replied with a cheeky comment but I meant to follow up: clojure inherently doesn't have generics and while I don't want to speak for the metosin folks, i suspect that the concept of "generics" fundamentally doesn't align with schema systems like malli. you can already achieve generics by just using :or and normal clojure functions: [:or (mapv #(do [:=> [:cat %] [:vector %]]) [number? keyword? string?])]#2022-06-0615:13Noah Bogarthaving said that, iwould be interested to see what you could build in this space as i think it has potential. it's cool to see the overlap between clojure schema systems and typescript's structural typing#2022-06-0616:10Ben SlessSimple schema can take an argument, make it the type#2022-06-0616:12Ben Slesshttps://github.com/metosin/malli#content-dependent-simple-schema#2022-06-0712:23Tiago Dall'Ocavery nice#2022-06-0712:23Tiago Dall'Ocamalli positively surprising me as usual#2022-06-0712:23Tiago Dall'Ocathank you all for the excellent work!#2022-06-0612:28Tiago Dall'OcaI'm developing mali-ts as a way to integrate malli schemas into typescript's type system and it'd be really good to have generics, though I understand why in clojure it might not make so much sense#2022-06-0612:29Tiago Dall'OcaMaybe it's not even that hard to implement? I'm not sure#2022-06-0612:31Tiago Dall'OcaAnd on the topic typescript integration, it'd be really good to have a way to model Promises and async stuff#2022-06-0616:13Ben SlessIf you have generics modeled then promises are solves, no?#2022-06-0612:31Tiago Dall'OcaIf doesn't make sense to have those in malli, I might implement in malli-ts only#2022-06-1219:55borkdudeHi! Can this be fixed somehow?
------ WARNING #1 - -----------------------------------------------------------
File: ~/.m2/repository/metosin/malli/0.8.4/malli-0.8.4.jar!/malli/core.cljc:2265:26
--------------------------------------------------------------------------------
2262 | (reduce -register-var {}))))
2263 |
2264 | (defn class-schemas []
2265 | {#?(:clj Pattern, :cljs js/RegExp) (-re-schema true)})
--------------------------------^-----------------------------------------------
References to the global RegExp object prevents optimization of regular expressions
I'm compiling malli as part of #nbb and I'd like to keep the rest of nbb optimized#2022-06-1220:18borkdudeFYI: https://github.com/babashka/nbb#metosinmalli#2022-06-1220:19juhoteperihttps://github.com/metosin/malli/pull/692 Needs new release#2022-06-1220:36borkdudeAwesome thank. cc @ikitommi#2022-06-1304:47ikitommiawesome! I’ll try to cut a release today (lot’s of small changes waiting)#2022-06-1308:08ingesolI tested function instrumentation in CLJS recently, and it works pretty well with the pretty printer. One thing I noticed as a pain point was that I wanted to see the input args when an output value does not match the schema. Would that make sense to add? Or did I simply miss it?#2022-06-1309:01ikitommiit’s not passed to the error reporter, should be a small change: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2453#2022-06-1309:02ikitommiissue and pr would be most welcome.#2022-06-1309:02ikitommino, the args are passed in#2022-06-1309:03ikitommijust not used: https://github.com/metosin/malli/blob/master/src/malli/dev/pretty.cljc#L51-L58#2022-06-1321:10ingesolThanks. Was a quick fix, posted a PR#2022-06-1605:01pinkfrogHi, is there some tool like https://github.com/Provisdom/spectomic but generates datomic schema from malli instead of spec ?#2022-06-1605:39Ben SlessI tried writing one once, I guess you could port some ideas from here and write a transformer like there is for json schema#2022-06-1607:37pinkfrogAre you saying porting ideas from the above repo?#2022-06-1607:48eskosI don’t think Malli’s transformers can do 1-to-n replacements (eg. single value to map), although I haven’t tested the behavior for such case.
That said, I don’t think it should be that hard to create equivalent behavior with some helper functions and whatnot, plus one area of malli I haven’t explored is the raw walk api, which may work just fine for this 🙂#2022-06-1608:54juhoteperiShould work. E.g.:
[:map {:decode/string (fn [s] (if (string? s) (js/JSON.parse s) s))} [:foo :keyword]]
Decode function for the map schema is run before checking map properties, and decoding those.
At least I've used this in cases where API takes comma separated string, and it is decoded to seq/vec and then items are coerced to keywords or such.
Not sure if this answers the original question, but 1-to-n transfomer tranformations should work.#2022-06-1608:55juhoteperi(decoding is one use/type of transformer)#2022-06-1611:49eskosOkay, that’s great. Tangentially, I just did a Thing™ which I didn’t mean to but can be abused to do something like this 😅 I’ll create another thread about it since it’s sort of a pre-release now.#2022-06-1611:53eskosThere’s that post. So yeah, https://github.com/esuomi/muotti/ can be abused like so:
(require '[malli.core :as malli])
(require '[muotti.core :as muotti])
(require '[muotti.malli :as mm])
(malli/decode
[:string]
1
(mm/transformer (muotti/->transformer
{:transformations {[:int :string] {:transformer (constantly {:a 1 :b "wow"})}}})))
Obviously custom schema registry support needs to be added to make this even remotely sensible :rolling_on_the_floor_laughing:#2022-06-1611:51eskosJust released https://github.com/esuomi/muotti/, a graph-based value transformation chain library which has integration with malli’s transformers.#2022-06-1611:52eskosMalli support is still somewhat in progress as malli has tons of schemas and some can be considered quite subjective (eg. for value transformation, what does ) but maybe those can be supported eventually as well. For now core types are supported.#2022-06-1611:54eskosAnd this is a sort of pre-release for now which is why I put this here, not #announcements - have to let the dust settle first 🙂#2022-06-1612:19ikitommiLooks great! like byte-streams, but for (literal) data. Malli will have effective and derived types, will simplify things a lot, https://github.com/metosin/malli/issues/264#2022-06-1612:26eskosThanks! 😸
Figuring out which schemas to support and how is going to be an on-going process, I think - I’m going to exclusively avoid doing validation with muotti, naturally. My intention is to start using this in an actual project quite soonish, so I’m hoping that’ll let me refine the native support and providing sensible defaults.
And as curiosity, muotti was made almost entirely during a few train trips between Helsinki and Tampere… 🙂#2022-06-1801:35ValentínHi, I want to use malli for form-input validation... I want to display customs errors#2022-06-1801:36ValentínHow could I display error msg base on the input value?#2022-06-1801:39ValentínExample: If the user did not type anything at all, I want to display "This field is required", is the user typed just one character, I want to display the following error msg "It should be at least 2 character long"#2022-06-1801:41Valentín(def form-name [:string {:min 2
:fn {:error/fn '(fn [{:keys [value]} _]
(if (empty value)
"This field is required"
"It must be at least 2 character long"))}}])
(def form [:map [:name form-name]])
(me/humanize (m/explain form {:name ""}))
; The behaviour I want
; => "This field is required"
(me/humanize (m/explain form {:name "J"}))
; The behaviour I want
; => "It must be at least 2 character long"#2022-06-1808:34George PeristerakisI think your error is in your precondition, the function you should use is empty? instead of empty#2022-06-1808:43Ben SlessI think the defaults already give you this behavior. Try deleting the fn property#2022-06-1814:58ValentínThere is also something wrong, the fn function never evaluate. @UK0810AQ2 I want to custom the default error msg.#2022-06-1815:42Ben SlessThe property should be :error/fn directly, without the preceding :fn key
But what's wrong with the default errors?
:string {:error/fn {:en (fn [{:keys [schema value]} _]
(let [{:keys [min max]} (m/properties schema)]
(cond
(not (string? value)) "should be a string"
(and min (= min max)) (str "should be " min " characters")
(and min max) (str "should be between " min " and " max " characters")
min (str "should be at least " min " characters")
max (str "should be at most " max " characters"))))}}
And if it's in a map you'll get "missing required field blah" from there
Also, I think you should use empty? and not empty#2022-06-2003:55Valentín@UK0810AQ2 did you run this code on the repl? I coud not make it works.#2022-06-2003:58ValentínAlso, I copy and paste the following code from github, and I'm getting the following exception... What am I missing?#2022-06-2003:58ValentínThanks#2022-06-2003:59Ben SlessI copied this example from malli's source, I assume it works#2022-06-2420:44jprudent@U0103KEKSLR you need to add the sci dependency to your project if your functions are quoted#2022-06-1801:42ValentínThanks in advance#2022-06-2004:06PanelCould it be done to validate with an async fn ?#2022-06-2016:52ikitomminot sure what is the question in here.#2022-06-2012:27borkdudeHow do you express [s/Long] (vector of Long in Schema) in malli?#2022-06-2012:32ikitommia [:vector :int] is close, but for all int?s.#2022-06-2012:32ikitommifor exact class, many ways (could be simpler), but one being:
(def a-long (m/-simple-schema {:pred #(instance? Long %)}))
(m/validate a-long (Integer. 12)) ;=> false
(m/validate a-long (Long. 12)) ; => true#2022-06-2012:33ikitommiwith min, max & json-schema translation:
(def a-long
(m/-simple-schema
{:type :long
:pred #(instance? Long %)
:json-schema {:type "long"}
:property-pred (m/-min-max-pred nil)}))
(m/validate [a-long {:min 10}] (Long. 12)) ; => true
(m/validate [a-long {:min 10}] (Long. 8)) ; => false#2022-06-2012:34ikitommisomeone asked full support for using Java Classes as Schemas. But that would not work with cljs.#2022-06-2012:34ikitommiwould allow [:vector Long]#2022-06-2012:35borkdude@U055NJ5CC The reason I'm asking is that my new (work-in-progress) CLI library is using a notation similar to schema for coercion:
https://github.com/babashka/cli
E.g.: {:coerce {:a :int}} will turn "--a" "1" into {:a 1} and {:coerce {:a [:int]}} will turn "--a" "1" into {:a [1]} . This coercion syntax doesn't have to be as powerful as malli. The idea is that you can coerce command line args into a map and then do validation on that (in your clojure function) using spec or malli (since it's a detail whether you are calling this function from the command line or from the REPL)#2022-06-2012:39ikitommiyeah, that looks great!#2022-06-2012:40ikitommithere is the malli-lite syntax too.#2022-06-2012:40ikitommi(defn coercer [schema]
(m/decoder (lite/schema schema) (mt/string-transformer)))
((coercer {:a :int}) {:a "123"})
; => {:a 123}#2022-06-2012:47ikitommiso, you will have your own transforming part there (the simplest way to do this, looks simple), but the future processing would be done using a full(er) schema/spec lib?#2022-06-2012:48borkdudeThe idea is inspired by clojure -X, you just call a clojure function from the command line basically. And whether you call this function via the command line, or via the REPL, the validation of arguments needs to happen anyway. So why put the validation logic in a CLI library, while it should probably live inside your app code#2022-06-2012:49borkdudeAnd so yes, you should do validation of args the same way you were doing it anyways in Clojure (with malli, schema, manual asserts)#2022-06-2012:49borkdudeAnd you can then choose whatever you were using. The main job of the CLI library is to transform strings into data, not much more than that#2022-06-2012:52borkdudeBy keeping it simple, you can also put your coercion "spec" in the deps.edn file (it doesn't require any function symbols, just keywords and collections)#2022-06-2012:55ikitommi👍#2022-06-2012:56ikitommithat’s one of the goal of malli too, to give a literal notation for schemas. Using full malli is too big for this case I guess? or does it miss something?#2022-06-2012:57borkdudeI want to it be an un-opiniated library so it works together with malli, spec, schema or your hand-rolled things#2022-06-2012:58borkdudealthough we could add some docs on how you can integrate with malli#2022-06-2012:58borkdudethe malli-lite syntax seems great for that#2022-06-2012:58ikitommiI think that’s a good choice. Is it possible to make it pluggable?#2022-06-2012:58ikitommiyes, happy to provide the example/glue if there is an extension point for that.#2022-06-2012:59borkdudewe could maybe do this using a protocol or multimethod? :thinking_face:#2022-06-2013:00borkdudebut even without this, it's easy to plug in your own thing, since after coercion you're dealing with just clojure data#2022-06-2114:42ikitommiwithout yet looking at the code, multimethod sounds good. Global Side Effects For The Win! 🙂 will check the repo later. thanks for the lib(s), again.#2022-06-2023:44PanelIf I have a form and want to validate on things that can only be done on server.
Could I validate a schema with an async call ?#2022-06-2101:43Lucy WangWhy not? Simply send the raw form data to the server using ajax, the server validates it, then send back the result.#2022-06-2103:04PanelI was wondering if this can be describe as a malli schema, maybe using a Fn schemas.#2022-06-2209:03Setzer22Is there a default schema type for dates? Something like :date or :inst (except I tried those but they don't work)#2022-06-2214:56eskosinst? predicate is supported, but not much else probably due to cross-platform support being tricky#2022-06-2210:31ingesolhttps://clojurians.slack.com/archives/CHY97NXE2/p1655893717482109#2022-06-2310:40ikitommifinally had time to release it, the pre-summer-holiday-edition 🌴#2022-06-2321:36devnany chance at compojure-api + malli? is that work that just needs doing or is there a fundamental reason why malli couldn’t be added as an option on compojure-api?#2022-06-2418:13devnlooking now, i suppose the answer is to use reitit#2022-06-2418:13devnbut either way i’m curious if a patch would be accepted to handle malli coercions in compojure-api#2022-06-2716:19Ben Sless@valtteri moving discussion of malli here, regarding json schema, the first item we should tackle, and hardest, IMO, is time schemas. There was a PR I opened and a long thread on it in the issues.
It seems like the biggest tension there is good-enough vs. specification compliance
How do you want to handle it?#2022-06-2716:22Ben Slesshttps://github.com/metosin/malli/issues/501#2022-06-2716:22Ben Slesshttps://github.com/metosin/malli/issues/49#2022-06-2716:27dvingofor cljs I would vote for https://github.com/henryw374/cljc.java-time (what tick uses) but how do you deal with this now being a dependency of malli?#2022-06-2716:27dvingoperhaps utilizing https://github.com/borkdude/dynaload ?#2022-06-2716:33valtteriHmmm personally I’d like to minimize dependencies#2022-06-2716:34valtteriI think the java.time stuff looks good and in CLJS we could perhaps just use js dates (I know, they’re bad) or goog.date?#2022-06-2716:35valtteriWe can always make initial release under malli.experimental.something so we can move on and get feedback#2022-06-2716:36dvingohow about a code path branch using dynaload? - one set of schemas backed by js dates and one by js-joda (cljc.java-time)#2022-06-2716:36valtteriAnd if it’s a decision between “good enough” and “specification compliance” this one leans to “good enough”#2022-06-2716:37dvingobut yea not a big deal because a date lib can always be added in user-space#2022-06-2716:39valtteriWould these be realistic goals?
• good and practical defaults for Java
• “good enough” defaults for JS
• no new deps
With a quick glance Ben’s suggestion for the java impl looks good and practical to me. I’ll promise to read through all the linked references tonight.#2022-06-2717:01Ben SlessThanks
I'd also try getting feedback from Henry, both on our current options and maybe on a path forwards, how we could reach full spec compliance while delivering something that covers 99% of use cases in the meanwhile#2022-06-2717:20valtteriOh, so you have a hard requirement for the full spec compliance?#2022-06-2717:22valtteriI think we anyway want to have extension points so that the behaviour can be customized.#2022-06-2717:23valtteriBut have defaults that work for the 99%#2022-06-2718:44Ben SlessWith an eye towards an end goal of using json schema or async api across an organization, with a variety of languages, I want to at least have parity in features#2022-06-2813:59Colin P. HillUnsure if this is in the scope of your discussion, but I'm currently working on an effort that uses the support Malli has for this so far, and besides a lack of built-in support for types beyond date-time, the other pain point I've been running into for time types is that the generated JSON schemas are less restrictive than the Malli schemas they're based on. A uuid? in Malli becomes a value with a string with a format of date-time in JSON – but JSON Schema validators are not required to treat formats as normative. This means another system could consume the JSON Schema output to validate a message, pass the message to the system using Malli, and then have it rejected as invalid. My workaround has been manually adding a pattern field.#2022-06-2917:08valtteriI've been thinking this and one way to circumvent the date/time type problem could be to treat them as strings. There could be a regex schema according to the format in JSON-schema. Dunno if this would make any sense though. But this is how it works in JSON-schema.#2022-06-2917:10valtteriI've read through all the earlier conversations and checked a couple of different JSON schema implementations how they do it. I think I'm more confused now than in the beginning. 😄#2022-06-2917:13Ben SlessWe could always ask in their slack#2022-06-2917:14valtteriYeah. But what do you think @UK0810AQ2 if in JSON-schema temporal types are strings with some enforced format.. would it make sense to do it in the same way on malli side?#2022-06-2917:15valtteriSo we could be compliant and move on 😉#2022-06-2917:15valtteriAnother thing we should clarify is that which version(s) of the JSON-schema spec are we targeting to support#2022-06-2917:26valtteriWe can also try to not solve the malli temporal schemas in general but make json-schema specific variants. If this would help us to move on#2022-06-2919:18Ben SlessThing is, in malli you can accept strings and make sure they transform to timestamps. When they leave the system they'll be serialized back to strings anyway#2022-06-2919:19Ben SlessThe temporal thing has become my bugbear#2022-06-2919:34valtteriYeah I understand that but I was thinking since timestamps are the “blocker” because there are so many possible ways to do them and it's difficult to make it good for both java and js, we could try to avoid those decisions.
Java side is more clear to me. We could also think “java first and js some day”.#2022-06-2919:36valtteriIf we think only Java and assume your suggestion about the types, there are no blockers…right? :thinking_face:#2022-06-2919:44Ben SlessNot from me, Tommi or Henry might have reservations we should account for#2022-06-2919:50valtteriI think we could whip up a working example to make it concrete. I guess the types can easily be changed if better suggestions pop up#2022-06-2919:51valtteriI mean a working example in context of json schema#2022-06-2919:52valtteriIt's easier to get that feedback when people can see it in action#2022-06-3018:13valtteriOh man, I just noticed that the old json-schema->malli PR is actually alive again!#2022-06-3018:18valtteriI don’t understand how I had missed that. I’ll comment to the PR that we’re thinking about these same things here. Dunno if https://github.com/tangrammer is in this Slack?#2022-06-2800:35DerekCross-posting from #announcements :
Announcing org.passen/malapropism. A small library for configuration data backed by malli
https://github.com/dpassen/malapropism#2022-06-2803:04pinkfroghttps://www.metosin.fi/blog/high-performance-schemas-in-clojurescript-with-malli-1-2/ Do we have a second series?#2022-06-2804:50Alexander MoskvichevIs there an easy way to change default date format in malli.transofm? I've read several discussions about custom registry, etc, but just started with malli, it's too hard for now to understand. I use malli with reitit, just need to change outgoing date format#2022-06-2817:47Setzer22Does malli support custom function predicates? I noticed some functions seem to be supported but most of them give an "invalid schema" error and I don't know if there's another way of doing this#2022-06-2817:48Setzer22actually, what I need is define a schema for a map that contains a core.async channel. I can always use :any but it would've been nice to use something like #(instance? ManyToManyChannel %)#2022-06-2817:54Setzer22Also, unrelated question 😄 I'm pretty curious about this: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#tldr
In particular, in this snippet:
(defn plus1
"Adds one to the number"
{:malli/schema [:=> [:cat :int] :int]}
[x] (inc x))
What's this way of using metadata to define the schema of the function via defn? Is this a defn-like macro defined somewhere?#2022-06-2817:58skynet@jsanchezf if I understand, it's the normal Clojure defn and that's the attr-map? argument which becomes metadata https://clojuredocs.org/clojure.core/defn#2022-06-2817:59Setzer22@craigy But then, how does it work? Malli has no way of knowing when a function is redefined. Simply defining some metadata does not lead to instrumentation happening#2022-06-2818:00dvingo(dev/start! {:report (pretty/reporter)})
check the source of that call for the answer 🙂#2022-06-2818:01dvingothis section https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defining-function-schemas explains it#2022-06-2818:01dvingo> Without instrumentation turned on, there is no schema enforcement:
#2022-06-2818:01dvingo> defn schemas can be defined with standard Var metadata. It allows defn schema documentation and instrumentation without dependencies to malli itself from the functions. It's just data.
#2022-06-2818:04Setzer22I see, so looking at the source (https://github.com/metosin/malli/blob/b745b73a93109a71643ce58302189a6f69c56d5e/src/malli/dev.clj#L15) it walks all the namespaces at the time where you call dev/start! , then looks for instrumented functions.
But how will it know when I create a new function or define a new namespace? Do I need to call start! again for it to pick it up?#2022-06-2907:25ikitommiyes, you need to call start! again when you annotate new functions. Tried to hook Var-watching for already defined schematized defns, but the Clojure core doesn’t support that easily. Calling start! makes the collecting explicit.#2022-06-2821:42ingesolI’m using malli to instrument functions and print results using the pretty printer. Some functions accept and return huge maps, so huge that it makes the output of the pretty printer unusable. Any hints on how to improve this?#2022-06-2906:27eskos• https://github.com/greglook/puget can make pretty prints even more prettier; sometimes just adding color helps a lot
• https://github.com/lambdaisland/deep-diff2 (or clojure.data/diff) can be used to diff the content; with a bit of trickery you can print only the interesting parts#2022-06-2907:21ikitommi@U8SFC8HLP Puget in nice, but Malli uses a custom pretty printer directly on top of fipp - different opinions about colors & works with CLJS too. About the huge maps, currently there is no omitting of valid values (https://github.com/bhb/expound#show-valid-values), would have needed that too, but have had no time to implement. PR would be welcome on this. Also, I think it’s the final piece before extracting the pretty printing from reitit & malli into a clean and minimal lib (https://github.com/metosin/virhe).#2022-06-2907:25ingesolI think showing valid values is useful to get the full picture. It also makes the report easier to parse sometimes. If there are 4 args to the fn, and 1 is wrong, it can be easier on the eyes to navigate to the wrong one when you use the valid ones as guides.#2022-06-2907:27ikitommiok, I’m all ears on how to make it better then.#2022-06-2907:27ingesolIn my case, I have a re-frame app-db. The number of keys in the map is not huge, but the number of nested keys is. So some way of sniffing out the approximate size of the map and deciding to print the first 500 chars or something would be nice. That was what I was looking for in my question. Would be nice if fipp had that option.#2022-06-2907:28ingesolFrom reading the code, it looks like we just pass the map to fipp for printing?#2022-06-2907:29ikitommihttps://github.com/metosin/malli/blob/master/src/malli/dev/virhe.cljc#L34#2022-06-2907:29ingesolgroup hug#2022-06-2907:30ingesolbut virhe isn’t currently used in malli, is it? I suppose maybe I can use it to implement an alternative to the default pretty printer?#2022-06-2907:30ikitommivirhe is… inlined 🙂
(ns malli.dev.virhe
"initial code for "
#2022-06-2907:30ingesolso… what I’m asking for is already available?#2022-06-2907:31ikitommiactual virhe-repo is just a README, “copy code here when it’s ok”#2022-06-2907:31ikitommiyes#2022-06-2907:31ikitommiyou can either swap the EDNPrinter or pass options to it to work differently#2022-06-2907:31ikitommiall code is in malli repo.#2022-06-2907:34ingesolawesome, thanks! I just didn’t notice that part of the API. Will have a go at this 🙂#2022-06-2909:26ingesolIn case it’s useful for anyone, here’s the change I needed. I wanted to avoid printing the re-frame app-db. Both because the contents are usually not interesting, and because it’s way too large to be printed effectively. In the case of a value not matching the schema of our app db, it would usually be something that does not match the check here so it would be printed anyway.
(extend-type v/EdnPrinter
fv/IVisitor
(visit-map [this x]
(if (:some-key-always-in-our-db x)
;; Avoid printing app db to console, as it is way too large
(v/-color :text "{... app db ...}" x)
;; Inlining original implementation from EdnPrinter
(let [xs (sort-by identity (fn [a b] (arr/rank (first a) (first b))) x)]
(fe/pretty-coll this (v/-color :text "{" this) xs [:span (v/-color :text "," this) :line] (v/-color :text "}" this)
(fn [printer [k v]]
[:span (fv/visit printer k) " " (fv/visit printer v)]))))))#2022-06-2822:10steveb8nmaybe #portal?#2022-06-3014:43stathissiderishello, does anyone know what the :schema schema does?#2022-06-3015:17ikitommiit's an eager reference, works like indentity. You can for example escape a sequence schema with it: [:cat :int [:* [:schema [:cat :int]]]#2022-06-3015:18stathissideristhank you! I thought you were on vacations 😉#2022-06-3018:06borkdudeA PR which bridges the gap between malli and #babashka!
https://github.com/metosin/malli/pull/718#2022-07-0109:41Eric DvorsakIs there a Malli equivalent to clojure.spec/every? I'm trying to write this spec in Malli https://github.com/clojure/core.specs.alpha/blob/master/src/main/clojure/clojure/core/specs/alpha.clj#L39#2022-07-0109:57valtteriHmmm I'm not sure if that exists directly but I guess it can be implemented with :and ?#2022-07-0112:20Eric Dvorsakyeah that's what I'm trying but can't really figure out how to achieve the same thing.
every allows to name and declare specs for different key/values in the map as tuples#2022-07-0111:16borkdude#2022-07-0403:41kokonutDoes malli have anything equivalent to Union in Typed Racket?
https://docs.racket-lang.org/ts-guide/beginning.html
#lang typed/racket
(define-type Tree (U leaf node))
(struct leaf ([val : Number]))
(struct node ([left : Tree] [right : Tree]))
It is the same concept with F#'s discriminated union.
I read malli documentation carefully but couldn't find.#2022-07-0404:29Ben SlessIsn't this just or?#2022-07-0408:00hansbuggeOr :multi which has support for discriminators via :dispatch
https://github.com/metosin/malli#multi-schemas#2022-07-0408:51Ben SlessIs the racket Union discriminating?#2022-07-0410:09kokonut@UK0810AQ2 I believe you can say that.#2022-07-0410:15Ben Slessmulti can dispatch to the correct schema which would make it discriminating. If you don't have a way of doing that, you can use or or orn, which is a named union#2022-07-0409:39Eric DvorsakI'm still trying to spec clojure.core with malli https://github.com/clojure/core.specs.alpha/blob/master/src/main/clojure/clojure/core/specs/alpha.clj#L39
I'm trying to find a way to replace this every which specs a map as 3 different kinds of key/values.
unfortunately Malli map-of takes only one kind of key
I'm wondering if there is a way to transform the values in the schema, so that it checks that it's a map then transforms in a repetitions of tuples.
Is it possible with Malli? going through the docs I can't find any clear way#2022-07-0410:03Eric DvorsakI think I found my solution in malli source https://github.com/metosin/malli/blob/2398df55ee806e25592fabf4d0c642ee3a2b233f/src/malli/destructure.cljc#L19#2022-07-0410:19Eric DvorsakIs it expected that the output of m/parse can be lossy?
(defn -map-like? [x] (or (map? x) (and (seqable? x) (every? (fn [e] (and (vector? e) (= 2 (count e)))) x))))
(def MapLike (m/-collection-schema {:type 'MapLike, :empty {}, :pred -map-like?}))
(m/parse [MapLike [:orn
[:some-keyword [:tuple :keyword :any]]
[:some-int [:tuple :int :any]]]]
{:k1 1
:k2 2
3 4
5 :six})
{:some-keyword [:k2 2], :some-int [5 :six]}
In this snippet above one can see that because the output of parse is a map, keys of the same tuple are overwritten#2022-07-0500:22Stig BrautasetHow to model a deck of cards?
We use spec at work, but I wanted to try Malli for a personal toy project. I promptly ran into a problem I couldn't solve, and I've tried to make a simplified version of it here.
Imagine a simplified game of poker: a hand is 5 cards, and we only care about whether it is a flush (i.e. 5 cards of the same suit) or not.#2022-07-0500:23Stig BrautasetFirst attempt
As we don't really care about the numbers or faces we can model a card as simple enum of its suit, and a hand as a 5-tuple of cards.
The deck is trickier. It has up to 52 cards, but I can't find a way to model the additional constraint that we should only have a maximum of 13 of each suit. Thus the last test in this self-contained example (usually) fails:
(ns cards.simple-test
(:require
[clojure.test :refer [deftest is testing]]
[malli.core :as m]
[malli.generator :as mg]))
(def Card [:enum :spades :hearts :clubs :diamonds])
(def Hand [:tuple Card Card Card Card Card])
(def Deck [:sequential {:max 52} Card])
(deftest deck
(testing "we can generate a Deck"
(doseq [deck (mg/sample Deck)]
(testing "that validates"
(is (true? (m/validate Deck deck))))
(testing "where every element is a card"
(is (every? #(m/validate Card %) deck)))
(testing "of maximum 52 cards"
(is (>= 52 (count deck))))
(testing "where each suite occurs at most 13 times"
;; this test (usually) fails, because we haven't restricted
;; the count of each suit to maximum 13.
(is (>= 13 (->> deck
frequencies
vals
(reduce max 0))))))))#2022-07-0500:24Stig BrautasetSecond attempt
Let's give each card a suit and number to address the problem with our previous deck.
A hand is now slighly more complicated: we have to use a set with {:min 5 :max 5} properties instead of a 5-tuple to ensure cards are not duplicated. (A hand with 5 ace of spades should not be valid!)
The deck, on the other hand, is simpler than before. By using a set we can drop the {:max 52} property. It is now implicit, because there are only 52 possible combinations of 4 suits and 13 numbers.
Self-contained example:
(ns cards.better-test
(:require
[clojure.test :refer [deftest is testing]]
[malli.core :as m]
[malli.generator :as mg]))
(def Card [:tuple
[:enum :spades :hearts :clubs :diamonds]
[:int {:min 1 :max 13}]])
(def Hand [:set {:min 5 :max 5} Card])
(def Deck [:set Card])
(deftest deck
(testing "we can generate a Deck"
(doseq [deck (mg/sample Deck)]
(testing "that validates"
(is (true? (m/validate Deck deck))))
(testing "where every element is a card"
(is (every? #(m/validate Card %) deck)))
(testing "of maximum 52 cards"
(is (>= 52 (count deck))))
(testing "where each suite occurs at most 13 times"
(is (>= 13 (->> deck
(map first)
frequencies
vals
(reduce max 0))))))))#2022-07-0500:25Stig BrautasetThis solves the problem with my first attempt, but introduces another: we can neither sort nor properly shuffle our deck now. 😞#2022-07-0500:34Stig BrautasetI think what I want is an ordered sequence of distinct values, but that may be because I can't see outside my Specs-shaped box. Is there an alternative way to model this with Malli?
(def Deck [:sequential {:distinct true} Card])
#2022-07-0516:26roklenarcicDefine card as tuple of suit and number up to 13. A deck is a distinct sequence of cards. Unshuffled deck is a sorted deck#2022-07-0516:37Stig BrautasetThat sounds like my second attempt:
(def Card [:tuple
[:enum :spades :hearts :clubs :diamonds]
[:int {:min 1 :max 13}]])
(def Hand [:set {:min 5 :max 5} Card])
(def Deck [:set Card])#2022-07-0516:38Stig BrautasetHow can I specify a distinct sequence? It seems you have to go to a set, as there’s no (documented) {:distinct true} property I can add.#2022-07-0518:57Stig BrautasetAh, I think I see. I just discovered :fn 😄#2022-07-0519:01Stig BrautasetThis appears to do what I want:
(def Card [:tuple
[:enum :spades :hearts :clubs :diamonds]
[:int {:min 1 :max 13}]])
(def Hand [:and
[:sequential {:min 5 :max 5} Card]
[:fn (fn [hand] (apply distinct? hand))]])
(def Deck [:and
[:sequential Card]
[:fn (fn [deck] (apply distinct? deck))]])#2022-07-0519:05Stig BrautasetThat can be made to work with my simplified version too:
(def Card [:enum :spades :hearts :clubs :diamonds])
(def Hand [:sequential {:min 5 :max 5} Card])
(def Deck [:and
[:sequential Card]
[:fn #(->> % frequencies vals (reduce max 0) (>= 13))]])
Thank you for the help 🙂#2022-07-0519:06Stig Brautasetduckie strikes again 🙂#2022-07-0501:59dumrathttp://malli.io not working?#2022-07-0717:58valtteriWorks for me :thinking_face:#2022-07-0616:39raymcdermottdoes anyone have experience / examples of using malli to generate openAPI docs that https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-from-example.html?#2022-07-0717:46valtteriIs there something special with ApiGW or should normal swagger 2.x work? There is malli.swagger namespace..#2022-07-0717:46valtteriOpenApi 3.x has been requested and it’s somewhere on the backlog but not under active development atm.#2022-07-0718:07raymcdermottI was thinking that the swagger ns would probably be OK but would be nice if somebody has already trodden the path. That's all really.#2022-07-0718:21valtteriPlease let me know how it turns out. 🙂#2022-07-0717:25bortexzHi, does malli support specifying the relationship between arguments and return on function schemas, similar to spec’s fdef :fn ? Couldn’t find anything related when going over the Readme#2022-07-0717:57valtteriHmmm I guess we don’t have that. :thinking_face: Feel free to file an issue on github. Also contributions are welcome!#2022-07-0809:56bortexzFound an existing one, will keep an eye on it 🙂 https://github.com/metosin/malli/issues/608#2022-07-0923:45PanelWould anyone has anything to share regarding generating Web Form from Malli ?
I have seen multiple conversations, blog post and even conf talk but none of those have code. Thanks#2022-07-1007:52eskosI suppose it would require a bit indirect and more involved route to approach, since having somesuch library would need to make assumptions about nesting, specific form events, usability issues (think tab navigation, grouping, aria tags...). So maybe the reason is that no one's just yet done a good enough system they're comfortable with abstracting to a library 🙂#2022-07-1008:08valtteriThis is something we're doing ad-hoc in different projects. We've discussed about wrapping the best practices into a lib and open sourcing it. But I'd say the status is still “cooking”#2022-07-1008:45PanelSounds like the interceptor pattern, everyone using it code a new implementation to fit their needs.
@U6N4HSMFW can you share describe how you do it ? Do you walk the schema and generate hiccup ? Do you have 2 data structures, one for the validation and one to describe UI specifics ?
One option I played with is to use a JS lib that accept json-schema, that did not work out for you ?#2022-07-1010:30valtteriJSON-schema is fine for simple cases and if that works for you then that’s a perfectly valid way to go in my opinion.#2022-07-1010:31valtteriWe’ve tried it both ways you described. Just a couple of months ago a colleague demoed “walking malli schema and generating hiccup” approach.#2022-07-1010:31valtteriI think the “generate ui from schema” approach pays off if you have really complex forms and a lot of them#2022-07-1010:32valtteriOr when you need to dynamically generate the forms#2022-07-1010:33valtteriI think you could also get quite far with something like Formik + malli validation#2022-07-1700:08geraldodev@U6N4HSMFW the malli piece is set on stone for me, as for the react library, could you please elaborate why that were choosen ? Was it the better at the time, has it a feature that more inline with the approach that you are using. I've started to look into react-hook-form https://github.com/geraldodev/react-form-hook-test/blob/use-form-state/src/main/app/core.cljs#L45 but I've stopped. Going back to it soon. Today my colleague pointed me to https://github.com/tannerlinsley/react-form , In the last two weeks I'm learning that code from tanner have high quality, I briefly looked today one example. It looked that it gets too deep into validation. It would be nice a library that interact with DOM (react) just enough and gives as the hooks (not react) points to bring malli awesomeness to our beloved clojure data strutures. I'm following this rabbit hole. Are you too ?#2022-07-1110:22Stig BrautasetSay I want a schema for an integer in the range 1-10 (inclusive). Is there a practical difference between these two? If so, which one is recommended?
repl> (mg/sample [:int {:min 1 :max 10}])
(2 1 3 1 7 4 7 4 10 3)
repl> (mg/sample [:and :int [:>= 1] [:<= 10]])
(1 1 1 2 5 1 9 6 6 2)
The former is shorter, and it's more intuitive to me to start with :int when that's the base type. But I don't know if there's a benefit to using :>= and :<= to enforce the bounds instead.#2022-07-1111:40valtteriI would use the one that reads better: :min :max#2022-07-1111:43valtteriAt least I'm not aware of any extra benefits. :>= and :<= compile a bit differently compared to :min :max
https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2281-L2290
vs.
https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L589-L598
and if I had to guess, I'd say the :min :max might be a little bit faster. But this is handwaving. 😉 And the difference is probably very small.#2022-07-1112:21Stig BrautasetThank you!#2022-07-1314:36George PeristerakisI have a registry with large data objects specs. I created a list of base specs to have a more consistent taxonomy ex: :pk-int [int? {:min 1 :unique true}] . What is the best way to represent these base specs? Should I have a separate registry or keep them in the same registry?#2022-07-1316:23DrLjótssonNewbie question: How do I specify that a value should be of a specific class, for example java.time.ZonedDateTime#2022-07-1316:51respatialized[:fn #(instance? % java.time.ZonedDateTime)]#2022-07-1316:52respatializedNot sure if there's another "official" way but I've done it that way in the past#2022-07-1316:58DrLjótssonThanks! I thought there might be a more official way but this will do just fine. #2022-07-1319:32Tiago Dall'OcaHey, more of a implementation question: why schemas instances are implemented with reify instead of using records? When I tried extending IPrintWithWriter to malli.core.Schema it didn't work because the type generated for schemas with reify isn't malli.core.Schemas#2022-07-1404:59LoicI have a question regarding key-transform and :map-of :
I have map keys that are list and I want to turn that list into vector of vector.
But I struggle to make a basic transformation work such as just turning keys into string in case of use of :map-of
;;work
(m/decode
[:map [:smth :int]]
{:a 2}
(mt/key-transformer {:decode name}))
=> {"a" 2}
;; does not work
(m/decode
[:map-of any? int?]
{:a 2}
(mt/key-transformer {:decode name}))
=> {:a 2}#2022-07-1511:04Ferdinand BeyerIs there a built-in way in Malli to strip away map keys that are not in the schema?
Let’s say I have an open map schema like this:
[:map [:person/first-name string?]
[:person/last-name string?]]
And a compliant value with an extra key:
{:person/first-name "Cosmo"
:person/last-name "Kramer"
:person/likes #{:fruit}}
I’d like to “sanitise” this data and only keep specified keys, recursively.#2022-07-1511:08Ferdinand BeyerThanks for acting as a sound stage. I found a simple way that can easily extended for recursive application:
(require '[malli.util :as mu])
(select-keys data (mu/keys schema))
#2022-07-1511:16valtteriThere's a built-in strip-extra-keys-transformer for value transformation https://github.com/metosin/malli#value-transformation#2022-07-1511:19Ferdinand BeyerAmazing, thanks! Missed that one 😊#2022-07-1511:19valtteriNo problem 🙂#2022-07-1913:15AJ JaroWe recently updated a functions schema definition and it wasn’t written correctly, but not caught during the tests. Is there a way that Malli can be configured to validate arguments during Clojure tests? It was caught during “manual” testing but not during the automated tests#2022-07-1918:52respatializedIn your test namespace:
(require '[malli.instrument :as mi]
[clojure.test :as t])
(defn instrument-fixture [f]
(mi/collect!)
(mi/instrument!)
(f)
(mi/uninstrument!)
)
(t/use-fixtures :once instrument-fixture)
#2022-07-2320:13hanDerPederThis works
(def ok-schema
{:thing/create [:map
[:status [:string {:default "a"}]]]})
I want to extract the status part into a reusable schema, so I do
(def bad-schema
{:thing/status :string
:thing/create [:map
[:status [:thing/status {:default "a"}]]]})
This is not valid. Throws an exception:
No implementation of method: :-into-schema of protocol:
#'malli.core/IntoSchema found for class: clojure.lang.Keyword
Hopefully it’s clear what I’m trying to do. Is this a bug or am I going about this the wrong way?#2022-07-2414:55pppaul[:string}#2022-07-2520:57hanDerPederhttps://github.com/metosin/malli/pull/729#2022-07-2414:58pppauli'm trying to edit malli.core, i can't figure out how to step through (get into a working cider debugging state) -map-schema. i tried pulling out the validctor method into it's own fn and setting that to cider-debug-defun-at-point but without success. any tips?#2022-07-2415:01pppaul(m/schema
[:map {:closed true}
[:optional {:optional true} [:enum :a :b]]
[:specified :int]])
code i'm using to try to trigger -map-schema#2022-07-2416:08pppaulI've also tried this, and i get a stack overflow error. i'm not really sure how to approach developing for malli.
(m/validate
(m/schema
[:schema {:registry {:map (-map-schema)}}
[:map {:closed true}
[:optional {:optional true} [:enum :a :b]]
[:specified :int]]
])
{
;;:hi "valid"
:specified 1
;;"bad" :error
}
)#2022-07-2416:09pppaulthis is where i have imported -map-schema into my ns and can eval it... i did this because of the stack overflows i'm getting when debugging. however, when i don't go into debug mode i don't get problems, when i'm in debug mode i get stack overflows
pretty wild#2022-07-2606:53Yehonathan SharvitI am surprised by how JSON transformer works
(m/decode :int "123" malli.transform/json-transformer) ;; => "123"
Why isn’t the string "123" converted to an integer?
In the same way,
(m/decode [:map [:foo :int]] {:foo "123"} malli.transform/json-transformer) ;; => {:foo "123"}#2022-07-2608:23vemvmalli.transform/-json-decoders doesn't have a string->double thingy
maybe you want to be looking at malli.transform/string-transformer instead?#2022-07-2609:24Yehonathan SharvitThe distinction between json-decoders and string-decoders is not 100% clear to me#2022-07-2609:29vemvMy impression is that string-transformer is for parsing string-based values, and json-transformer for enriching values from already-parsed json with JVM/clj types (uuid, date, keyword etc) not present in json
So it might make sense to compose string-transformer with json-transformer#2022-07-2610:42Yehonathan SharvitIt makes sense to me also#2022-07-2610:55Yehonathan Sharvit@U45T93RA6 Is there a transformer that keywordize map keys when the schema keys are keywords?#2022-07-2610:58vemvlet's move it 1:1#2022-07-2613:54pppaulI have a list of keyseqs [[:parent :child1 :child2]...]that map in a human readable way into my schema, but when i do mu/get-in mu/update-in the mappings don't work. when i do subschemas the paths are sorta like [:parent 0 0 :child1 0 0 :child2] and :parent will have 10 different paths to the same :child1 and :child2. it seems a bit like a flip of a coin if i get the correct path or not. i've tried a few different ways of figuring out the correct one (smallest count). also the :in and :path are different, and i'm not sure what to use in mu/update-in. unfortunately i'm not in control of the schema (it's generated based off of json-schema). my next approach will involve recursive mu/find-in, but i'm wondering if anyone has run into a problem like this before, where conceptual indexing and real malli indexing have a significant mismatch and how to get to the correct spots in the schema to programmatically edit it (i'm adding custom properties)#2022-07-2705:44Martynas MIs there a way to represent this as a map?
{id: string?
size1: my-schema1
size2: my-schema1
size3: my-schema1}
Where I know that id will be present all the time and size1 and others are dynamic.
i.e. they can exist or they can have completely different names.
I could use map-of for the second part but I would like to use map for the id part.#2022-07-2708:11Yehonathan SharvitInteresting question @U028ART884X#2022-07-2708:13Martynas MInstead I hardcoded the keys. It was simpler than think and solve for truly any key 😄#2022-07-2710:37pppaulyes, i posted an issue about something like this on malli's github#2022-07-2710:39pppaul(let [{:keys [additionalProperties]} form
req-fields (->> required (mapv keyword) (into #{}))
all-fields (concat props req-fields)]
(if-not additionalProperties
(make-map props req-fields)
[:and
[:fn (fn [m]
(->>
(apply dissoc m all-fields)
(m/validate (m/schema [:map-of 'string? 'string?]))))]
(make-map props req-fields)]))#2022-07-2710:39pppaulthat's code i use to transform a json schema map to act something like what you described#2022-07-2710:40pppaul[:and
[:fn (fn [m]
(->>
(dissoc m :optional :required)
(m/validate (m/schema [:map-of keyword? string?]))))]
[:map
[:required [:enum :a :b]]
[:optional {:optional true} [:enum :a :b]]
]]
ends up making something like that#2022-07-2710:42pppaulmake-map (fn [props req-fields]
(apply vector :map
(->> props
(mapv (fn [[k v]]
(if (get req-fields k)
[k v]
[k {:optional true} v])))
(sort-by first))))#2022-07-2710:45pppaulthere is some discussing in malli about making :closed be a schema, supporting this idea with the work around that i posted, but it's not got a PR or anything, so work around is best. you can replace the :map-ofwith a schema for each key very easily with the code i provided#2022-07-2819:36nonrecursivehi all! i’m using reitit with malli for parameter coercion, and I have a schema with default values for some parameters, but those default values aren’t being added. has anyone else run into this?#2022-07-2819:37nonrecursivemy router looks like this:
(def router
(rr/router routes
{:data {:coercion reitit.coercion.malli/coercion
:middleware route-middleware}}))#2022-07-2822:00pppaulthere is a default values transformer#2022-07-2822:13nonrecursiveI discovered the reason, it was because it had {:optional? true} in the options#2022-07-2820:54Noah Bogartis it possible to say "this is either a set or the specific keyword :bypass "?#2022-07-2820:55Noah Bogartrealized my mistake immediately, lol. the answer is: [:or set? [:enum :bypass]]#2022-07-2823:48Mateo DifrancoWhat would be the best way of handling multiple possibilities of a map inside a schema, based on a top level keyword?
Maybe I can explain myself better with some examples, let's say I have something like this:
{:something "test"
:type :a
:deep {:depends 3}}
and
{:something "test"
:type :b
:deep {:another [1 2]}}
So, based on :type, I'm expecting different things in :deep. This seems to be a good case for :multi, given it allows using a function for dispatch. But the dispatch function would take the :deep map, not the top level whole map.
I have a work-around for this, but it's quite ugly and I'm wondering if there's a better way.#2022-07-2902:02pppaulthis looks like what multi is designed for, i haven't used multi, i'm curious where it fails in this case.#2022-07-2914:32Mateo DifrancoWell, I might be wrong, but take a look at this:
(def Something
[:map
[:something string?]
[:type [:enum :a :b]]
[:deep
[:multi {:dispatch :type}
[:a [:map [:depends int?]]]
[:b [:map [:another [:vector int?]]]]]]])
(m/validate
Something
{:something "test"
:type :a
:deep {:depends 3}}) ;=> false
And if I do a really ugly side effect, we can look at what exactly the dispatch function takes:
(def Something
[:map
[:something string?]
[:type [:enum :a :b]]
[:deep
[:multi {:dispatch (fn [x] (or (println x) :type))} <-------------
[:a [:map [:depends int?]]]
[:b [:map [:another [:vector int?]]]]]]])
(m/validate
Something
{:something "test"
:type :a
:deep {:depends 3}}) ;=> false, but also prints {:depends 3}#2022-07-2916:32valtteriYou probably should do the multi for the parent of :deep#2022-07-2916:33valtteriIf you need to dispatch on the :type of the parent#2022-07-2916:36Mateo DifrancoI thought about that, but then how would I avoid duplication of the structure?
For example, if I move :multi to the top of the schema, I would need to make each possibility contain the whole structure right? In this case, :a would have a schema matching :something for example. Maybe I can merge in each possibility though, but I was wondering if there was a better solution.#2022-07-2916:48valtteriYep, I guess that's whats you gotta do. You can splice the sub-schemas into smaller chunks for reuse#2022-07-2916:49valtteriThere might be some better way I'm not aware of 🙂#2022-07-2916:17annarcanaare there any SQL libraries that work with malli?#2022-07-2916:35valtteriDepends on what you mean with that. :) We have experimental tool that derives malli schemas from sql table schema#2022-07-2918:03colliderwriteror this (no personal experience):
https://github.com/kwrooijen/gungnir#2022-07-2918:05annarcanaYes! This looks like what I had in mind (data-driven SQL schemas)#2022-07-2918:06annarcanaWe are already using malli to generate reitit routes from data models, so one thought was to also generate SQL table schemas from same#2022-07-2918:07annarcanaI suppose any data-driven library like gungnir could work but something that already can do malli->SQL type translation was something I was hoping would be out there somewhere#2022-07-2918:07colliderwriterafaict, this does not do schemas but it seems to have all the parts#2022-07-2918:10annarcanait can do table create/drop migrations which should be enough for my purposes, but I'll have to dig into it#2022-07-2918:12colliderwriteri'd be interested to hear your thoughts#2022-07-3014:36AlejandroHello. Is there a blog post, a documentation page, anything that discusses motivation behind malli? I'd like to know how spec, plumatic/schema, and malli relate to each other.#2022-07-3015:09valtteriHere’s a video where Tommi explains also the motivation https://www.youtube.com/watch?v=MR83MhWQ61E#2022-07-3015:10valtteriComparison and reasoning wrt other tools (schema, spec) starts around 3:30#2022-07-3015:35Alejandrothank you#2022-07-3014:37pinkfrogm/validate returns true/false, what’s the function that throws exception when data is malformed?#2022-07-3014:40pinkfrogThe purpose of doing this is, I want to add code like
(m/validate :int foobar)
inside a function, and when foobar is not int, I want an exception is thrown so I can know some thing bad happens in sentry.
Another good to have is, the m/validate (equivalent) can be turned on in dev and off in prod.#2022-07-3014:54annarcanaYou'll most likely want some combination of explain/`humanize`. (https://github.com/metosin/malli#error-messages) I don't believe malli has any function that deliberately throws an exception, by design, but you can always use throw with an Exception whose payload is the malli explain output.#2022-07-3014:56pinkfrog@annapawlicka Maybe I shall just do (assert (m/validate ,,,))#2022-07-3017:12Noah BogartIn the simplest case, this is what I would suggest doing.#2022-07-3015:12Ben SlessDo you want this with decode, too?#2022-07-3101:31pinkfrogyes!#2022-08-0315:54Ben Slessyou should build something like:
(defn coercer
[schema options transformer]
(let [schema (m/schema schema options)
validator (m/validator schema options)
decoder (m/decoder schema options transformer)
explainer (m/explainer schema options)]
(fn [x success failure]
(let [ret (decoder x)]
(if (validator ret)
(success ret)
(failure (explainer ret)))))))#2022-08-1504:37pinkfrog@UK0810AQ2 Say I do not want to define the validators in advance. So is it meaningful to create the validators on the fly or shall I just directly use the m/validate function? Is there any performance discrepancy regarding the two?#2022-08-1505:11Ben SlessYes, closing over a validator is faster than calling validate every time. In what context are you using this?#2022-08-1505:32pinkfrogAcross the codebase, many functions accepts a map but the data type etc is not clear. I want to annotate the functions.#2022-08-1507:01Ben Slessah, you should have said so, you want this: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schema-metadata#2022-08-1508:22pinkfrogUsing function schemas, there is a tendency to spec on all the arguments, but I’d rather spec on one of them.#2022-08-1508:24Ben SlessWhy not spec the others as any?#2022-08-1508:29pinkfrogThat causes confusion.#2022-08-1508:34Ben SlessI think you don't have to spec all the arguments with this macro, no?#2022-08-0305:02Stel AbregoHey y'all, I'm wondering if it's possible in Malli to create a spec for a map with these qualifications:
• There are some specific keyword keys that are required (this is simple in Malli)
• There can also be an arbitrary amount of keys that are vectors, but they all must be vectors of keywords and their values must be maps. (not straightforward)
• Besides the two possibilities above, the map is closed (ex no string keys for example, also not straightforward)
The last two things are the weird requirements. Seems like :map can't really support this. I might need to use [:and [:map ... ] [:fn ...] but even then it might be awkward to satisfy #3.#2022-08-0316:23pppaulusing :fn is how i did exactly this#2022-08-0316:24pppaulhttps://github.com/metosin/malli/issues/682#2022-08-0316:24pppaullast post is the solution, unless you want to give malli a PR and do some hard work#2022-08-0316:26pppaul[:and
[:fn (fn [m]
(->>
(dissoc m :specified :optional :required)
(m/validate (m/schema [:map-of keyword? string?]))))]
[:map
[:required [:enum :a :b]]
[:optional {:optional true} [:enum :a :b]]
[:specified :int]]]
make a function that outputs something like this#2022-08-0322:56Stel Abrego@U0LAJQLQ1 thank you for the help! I'm going to look at this later.#2022-08-0323:01pppauljust be aware that i didn't provide a general function, you need to create one that respects the keys in your malli map schema.#2022-08-0419:19Stel AbregoA little update, I realized that clojure.spec can solve this problem with actually great error messages so I might rewrite in spec. Might write a blog post about it later too.#2022-08-0315:52Ben SlessWhat's the best way to express in malli "a map must contain at least one of a set of kv"?#2022-08-0316:46Noah Bogartis that :or or :multi?#2022-08-0317:27Ben SlessIt's probably :or, but it's gross to have [:and M1 [:or M2.0 M2.1 M2.2]]#2022-08-0317:29Ben SlessI also think this requires some awareness of implementation details to get coercion right, because when dealing with two keys, you'll want to try to coerce both, i.e. [:or [:merge Ma Mb] Ma Mb]]#2022-08-0317:29Ben SlessThat's even messier and for only two keys#2022-08-0317:29Ben Slessnow do that with three keys, the combinations become unreasonable#2022-08-0317:35Ben SlessThe correct thing to do in these situations is compile an execution plan#2022-08-0317:35Ben SlessSort of like pattern matching libraries do#2022-08-0317:45Noah Bogartyou could also do a function schema for this, make the various keys optional and then perform the check in the :fn#2022-08-0318:06Ben SlessI am morally opposed to function schemas#2022-08-0318:19Noah Bogartany particular reason? seems appropriate for this kind of work#2022-08-0319:51Ben SlessThey're completely opaque, not well defined, not serializable, hard up to impossible to generate#2022-08-0319:52Ben SlessThe semantics of "at least one of this group" are clear, don't need a function. It can (should) be another schema type#2022-08-0319:58Noah BogartThat makes sense to me. I wonder how hard it is to write a new schema type like this#2022-08-0320:16Ben SlessA bad implementation is easy, an efficient implementation is hard 🙂#2022-08-0320:18Noah BogartIs it? Naively, I would expect it to be an or:
(fn [obj]
(or (contains? obj :a)
(contains? obj :b)
(contains? obj :c)))
or something similar. is there more to it?#2022-08-0320:18Noah BogartMaybe I'm misunderstanding the actual problem#2022-08-0320:21Ben Slessbecause there's a combinatoric explosion, you need for two keys:
[:or [:map [:k1 S1] [:k2 S2]] [:map [:k1 S1]] [:map [:k2 S2]]]#2022-08-0320:22Ben Slessfor three keys you have 7 options (I think, I'm tired)#2022-08-0320:22Noah Bogart"At least one of a set" means as long as one exists, it doesn't matter if others exist, so you can just check each one#2022-08-0320:23Noah Bogartcould compile a list of the ones that do exist and then merge them after the fact too#2022-08-0320:23Ben SlessYes, but now make decode, encode, parse and explain work 🙂#2022-08-0320:24Noah Bogarthah I won't be nerd sniped! but yes, if you don't want to directly implement a bunch of stuff, I think I can see the issue more clearly#2022-08-0320:25Ben SlessThat's why I dislike function schemas 🙂#2022-08-0423:58mauricio.szaboFolks, a simple question about .pretty/explain - is it possible to NOT print the full value being matched?
The reason is: I have a HUGE data, and it's actually breaking my terminal limit, so I can't actually see the error 😞#2022-08-0521:36ingesolhttps://clojurians.slack.com/archives/CLDK6MFMK/p1656494811406489?thread_ts=1656452534.761089&cid=CLDK6MFMK#2022-08-1006:57ikitommiwould be easy to add an option to drop that. please write an issue. Also, see https://github.com/metosin/malli/pull/731#2022-08-1006:57ikitommie.g. option (or new default) will mask the irrelevant parts of the value. they can be huge.#2022-08-0500:02pppaulyou can write it out to a file#2022-08-0500:07mauricio.szaboNot really, I'm on React-Native#2022-08-0501:02pppaulyou may have to edit some of the functions in the namespace to customize the output.#2022-08-0518:26Colin P. HillIn Malli function signatures, what does the ? prefix on a parameter mean?#2022-08-0519:08valtteriIt means that by convention the function will tolerate nil and possibly other nonsense for those params.#2022-08-0610:10hjrnunesI need to validate a map-like type with metadata. Actually, values are collections - vector-like types - which also have metadata that needs validation.
I've played around with the -map-schema and -collection-schema implementations and I can make them support my custom types easily, but I am unsure how I can express the metadata schema. I could use properties but I wonder if that makes sense. The metadata is as important as the rest. Actually it's where the most variation will be.
Is there a better way than simply maintaining two schemas - one for the actual type and another for the metadata map - and running validation for each independently?#2022-08-1006:48ikitommicurrently, there isn’t an extension point for validation, like there is for most other schema applications (transforming, generating, …), so there isn’t an easy way to add a generic, properties-based declaration for meta-data, if there was, could be something like:
[:map {:metadata [:map [:closed :boolean]]} [:x :int]]#2022-08-1006:49ikitommiif you can provide an example how and where you would like to define the metadata schemas, could suggest ways to build such thing. It’s definetely doable, many ways to do that.#2022-08-1006:50ikitommiPlease write an issue with some sample data?#2022-08-1007:12hjrnunesWell, that's the thing. I'm not really sure how to best handle it. I can easily do it via properties like the snippet you shared, but it just felt odd to have schema in the properties. Maybe I'm just overthinking it.#2022-08-1007:26ikitommicould also be a lookup, e.g. [:map {:domain/type ::object} …], so that the metadata-schemas are in a separate registry and looked out with a property-key. You can the create a helper that takes a schema and returns a metadata validator and use that when needed. keeps the validation of metadata and values as separate. should be easy’ish to do with schema walking.#2022-08-0621:36Eugenhi, is there a guide to learn malli for dummies?
I am new to clojure and I kind of struggle to get my head around malli.
I would like to use it for data modeling - as an exploratory tool.
I would like to define some entities and be able to customize the generated output.
(have a list of names, companies, etc).
I read through the section and I still feel like I am missing some context.
For example, I would like to generate company names like: Acme + some random smallish number ?!: ACME 12
I have defined an ID like
(def id [:string {:min 1, :max 20}])
and I would like to limit what characters it should contain (it should have only printable characters for example)#2022-08-0622:07hanDerPederMalli, as spec, leverages the test.check library for generating values. I would start there. There are some very useful guides linked in the readme of that project. #2022-08-0622:08hanDerPederhttps://www.youtube.com/watch?v=F4VZPxLZUdA#2022-08-0622:08hanDerPederThis one seems relevant#2022-08-0622:17Eugenthanks @U013U475882#2022-08-0622:19EugenI think my end goal is very similar to what @U07FP7QJ0 talked about here https://www.youtube.com/watch?v=ww9yR_rbgQs&ab_channel=LambdaIsland .
Thank you @U07FP7QJ0 for making the video !
I would love to have a more in-depth presenatation of that.
If you have anything please share.
BTW, can you share how the project evolved?
It's been two years.#2022-08-0813:46pppaul(def id [:string {:gen/elements [list of names to use] :min 1, :max 20}]) #2022-08-0813:47Eugenthanks for all the help, I am getting my head around it - but it's lots of new stuff to learn (test-check).#2022-08-0813:49EugenI am thinking of writing some generators that will leverage data sources like WikiData to get real world examples.
This should be more to generate some lorem ipsum style of data.#2022-08-0813:49pppaulif you aren't experienced with generators there is a bit of a learning curve, but it's not too hard. i find programmatically attaching the generators harder than making them.#2022-08-0813:50Eugenthanks for the feedback.
That is a question I have - how to use the generators in the app.
Since it's a lot of new stuff I am bound to make mistakes, hence me asking.#2022-08-0813:51pppaulif you have schema's that can generate nested things, you will want to control those generators a lot (any? is a very nasty one).#2022-08-0813:52Eugenif I make it I will be bale to write on CV: generator tamer 🙂#2022-08-0813:57pppaulalso, the gen/ properties don't work together, so if you want to use elments->fmap you have to use gen/gen instead. if you are converting to json, you need to watch out for symbol/string/keyword keys causing dupe key issues#2022-08-0813:58Eugendon't know what that means for now, but I'll try to keep an eye out for it.#2022-08-0813:58Eugenthanks#2022-08-0813:59pppaulmaybe not a normal issue to run into, but it was an issue for me when i was making generators for schemas that i converted from json to malli (i didn't own the schema)#2022-08-0814:01pppaul{'key :a :key :c "key" :d} that in json is a map where all keys are the same, and thus invalid#2022-08-0814:03pppaulanyway, biggest issue with generators is the recursive stuff, so if you have any of that you have to take care of it right away and figure out how to limit the recursions, or your data is going to get extremely large#2022-08-0814:04Eugenthanks#2022-08-0814:04EugenI'll try to stay away from recursion#2022-08-0816:44Noah BogartJust opened an https://github.com/metosin/malli/issues/734 for a subtle implementation detail/maybe bug with :tuple schemas: if given a seq, m/encode silently fails to do any encoding instead of coercing to a vector.#2022-08-0823:46Eugenis there a guide/version of this article with malli ? https://cognitect.com/blog/2017/3/24/3xeif9bxaom78qyzwssgwz1leuorh4#2022-08-1006:30ikitomminot exactly that, would you be interested in writing that?#2022-08-1006:32ikitommishould be easy to map that to malli#2022-08-1008:54Eugensounds interesting, but right now I don't have time for that. getting my bearings and it takes quite some time. I will consider this in a few months when I get more confient
I did (re)found specmonstah (the third time) https://github.com/reifyhealth/specmonstah/ .
On papaer it provides what I need so I am going to work with that and see if it's a good fit for what I am looking for: generate data for a web app to facilitate demos#2022-08-0923:52bortexzHi, recently started playing with Malli, I am trying to understand why this works:
(def Map1
(mu/optional-keys
(m/schema
[:map ::a ::b]
{:registry (merge
(m/default-schemas)
(mu/schemas)
{::a string?
::b [:merge
[:map [::a [:ref ::a]]]
[:map [:y string?]]]})})
[::b]))
but this fails with enum child error:
(def Map2
(mu/optional-keys
[:map {:registry (merge
(m/default-schemas)
(mu/schemas)
{::a string?
::b [:merge
[:map [::a [:ref ::a]]]
[:map [:y string?]]]})}
::a
::b]
[::b]))
=>
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1, :max nil}
Is it a bug? Am I missing something?
Edit: Another question that is slightly related (involves registries too) malli.util derivative transforms like mu/optional-keys obfuscate the refs when used inside registry:
(def Map1
[:map {:registry {::b string?
::a (mu/optional-keys
[:map [::b [:ref ::b]]])}}
::a])
=>
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/invalid-ref {:type :ref, :ref :malli-domain-lab/b}
Is there any workaround for this?#2022-08-1006:42ikitommiHi. I believe it’s a bug, please write an issue.#2022-08-1006:45ikitommiRobust way to define a schema space for an application is to use the global scope, e.g.:
(mr/set-default-registry!
;; linear search
(mr/composite-registry
;; default schemas
(m/default-schemas)
;; utils
(mu/schemas)))
(mu/optional-keys
[:map {:registry {::a string?
::b [:merge
[:map [::a [:ref ::a]]]
[:map [:y string?]]]}}
::a
::b]
[::b])#2022-08-1008:16bortexzDone! https://github.com/metosin/malli/issues/736
Thank you, I’ll rearrange how the registries are defined to make it work#2022-08-1007:02ikitommigetting back to computer this week from the long summer vacation, happy to see there the lively discussions here. if there is something clearly not resolved / answered by someone (🙇 for everyone for helping others!), you can mention me in the thread. will try to read the new new issues & prs this week.#2022-08-1010:50AkizHi, I need to change all keys in the schema from :keyword to :string.
Is there an utility for this?#2022-08-1011:54ikitommiwhat do you mean by keys here: map keys in values? or in schemas? something else? example would help#2022-08-1012:26AkizSure.
Here's an example. I would like to achieve that I don't have to define the output-schema manually.
In this case it would be enough to replace keyword with string and vice versa.
But It would be great if I could somehow use a transformer to transform only schemas.
(require '[malli.core :as m])
(require '[malli.transform :as mt])
(def input-schema [:map [:name :string] [:type :keyword]])
(def output-schema [:map [:name :string] [:type :string]])
(defn encode
{:malli/schema [:=> [:cat input-schema] output-schema]}
[val]
(m/encode input-schema val mt/string-transformer))#2022-08-1012:28ikitommiwould this help: https://github.com/metosin/malli/pull/737#2022-08-1012:29AkizYes, this is exactly what I tried unsuccessfully to achieve via schema-walk. Thanks!#2022-08-1012:30ikitommi👍#2022-08-1012:38ikitommiMerged the error-value helpers into master. In case of validation error, you can get just the values in error and/or mask the valid values with something like …, like expound does. Should be useful in pointing out (small set of) errors in huge values. feedback welcome, ping @mauricio.szabo:
(def Address
[:map {:closed true}
[:id :string]
[:tags [:set :keyword]]
[:numbers [:sequential :int]]
[:address [:map
[:street :string]
[:city :string]
[:zip :int]
[:lonlat [:tuple :double :double]]]]])
(def address
{:id "Lillan"
:EXTRA "KEY"
:tags #{:artesan "coffee" :garden "ground"}
:numbers (list 1 "2" 3 4 "5" 6 7)
:address {:street "Ahlmanintie 29"
:zip 33100
:lonlat [61.4858322, "23.7832851,17"]}})
(-> Address
(m/explain address)
(me/error-value {::me/mask-valid-values '...}))
;{:tags #{"coffee" "ground" ...},
; :numbers (... "2" ... ... "5" ... ...),
; :address {:lonlat [... "23.7832851,17"], :street ..., :zip ...},
; :EXTRA "KEY",
; :id ...}#2022-08-1012:39ikitomminext thing would be to integrate this into pretty explaning. I think it would be good to mask out valid values by default? and the options being:
1. mask valid values (default)
2. show full values
3. don’t show value at all#2022-08-1012:41ikitommithe tests also show how to do the same thing for individual errors. e.g. present errors one-by-one - like expound does.#2022-08-1013:15mauricio.szaboYes, I think masking valid values by default is good - it seems "prettier" this way, and makes for less noise IMHO#2022-08-1014:48ikitommihttps://github.com/metosin/malli/pull/738, feedback and testing most welcome:
(malli.dev.pretty/explain
[:map
[:id :int]
[:tags [:set :keyword]]
[:address [:map
[:street :string]
[:city :string]
[:zip :int]
[:lonlat [:tuple :double :double]]]]]
{:id "123"
:EXTRA "KEY"
:tags #{:artesan "coffee" :garden}
:address {:street "Ahlmanintie 29"
:city "Tampere"
:zip 33100
:lonlat [61.4858322, 23.7832851]}})
=>#2022-08-1018:07Mateo DifrancoAwesome!#2022-08-1109:34ikitommi^:-- if someone can show an real-life example where the value masking is still bad, please share (privately ok too). I guess if one has huge nested maps, masking each irrelevant key separately might still cause large values. Or a list of 100 entries with the last one in failure would cause 99 …s, which might not be best. For maps, all the irrelevant keys could be optionally accumulated into single entry on … as both key and value and for sequences, a …99 to show how many valids there are.#2022-08-1119:34dangercoderHi, great work on Malli! I’m trying to figure out how I can use malli without duplicating attributes. Is the “What I want” syntax possible in some way?
;; What I have
[:map
[::order-id :int]
[::order-multiplier :int]
[::order-amount decimal?]]
;; What I want
[:map
[::order-id]
[::order-multiplier]
[::order-amount]]
(m/validate ::order-id "123")
;; => false
#2022-08-1119:41valtteriYou can define a registry and register your schemas there. Here’s an inline example
(m/validate ::order-id "123" {:registry (merge
(m/default-schemas)
{::order-id :int
::order-multiplier :int
::order-amount decimal?})})#2022-08-1119:42valtteriMore info how to work with registries here https://github.com/metosin/malli#schema-registry#2022-08-1119:43dangercoderThanks a lot for the answer, cheers!#2022-08-1120:03dangercoderCool thing. I created a custom “register function”.
(s/register ::api-key [:string {:min 1}])
in IntelliJ/cursive i resolved my function as clojure.spec.alpha/def — this way I can jump to the schema-definition of a namespaced keyword if there is a register call. 😄#2022-08-1120:03dangercoder🤯#2022-08-1120:27valtteriCool!#2022-08-1510:20pinkfrogHi, running this example I got sci not available
(m/decode
[string? {:decode/string 'str/upper-case}]
"kerran" mt/string-transformer)
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/sci-not-available {:code str/upper-case}#2022-08-1510:21borkdudeI think you should include SCI as an optional dependency yourself. If you don't use the quotes, you won't need SCI#2022-08-1510:39pinkfrogThanks. Dunno what’s the added benefits of sci.#2022-08-1510:40borkdudeThe benefit of SCI would be if you would like to serialize your schemas to the front-end#2022-08-1510:40borkdudeThe quote is only needed for serialization#2022-08-1510:42pinkfrogThen why not a custom export function to convert function to its symbol names. I am not quite getting the actual benefits compared of importing sci here.#2022-08-1510:43borkdudeBecause just a symbol won't give you evaluation in the client#2022-08-1510:46pinkfrogWhen client are you referring to the js environment? Also, pardon me, are we assuming sci could run on js (given the recent work of nbb and such)#2022-08-1510:46borkdudeyes, JavaScript#2022-08-1510:47borkdudeyou can include malli on the client side and then use SCI to evaluate your schemas and get exactly the same validation as on the server#2022-08-1510:49borkdudeSo you can send your schema's dynamically to the client#2022-08-1510:51pinkfrog> So you can send your schema’s dynamically to the client
That is so futuristic, that I am not sure I am ready.#2022-08-1511:16valtteriIn the latest version of the README the examples have been changed to not require SCI unless mentioned.#2022-08-1511:17valtteriI think this particular one is now
(m/decode
[string? {:decode/string clojure.string/upper-case}]
"kerran" mt/string-transformer)
; => "KERRAN"#2022-08-1510:40pinkfrogI wonder why https://cljdoc.org/d/metosin/malli/0.8.9/doc/readme#value-transformation:~:text=Adding%20optional%20keys%20too%20via%20%3A%3Amt/add%2Doptional%2Dkeys%20option%3A. here uses qualified keyword, ::mt/add-optional-keys, but others like :decode/encode does not use qualfied ones#2022-08-1512:09ikitommiMalli started with simple keywords everywhere, but as the options are passed everywhere, this doesn’t scale well -> moved to use qualified keywords. Could clean up all options for 1.0.0?#2022-08-1512:11ikitommiOne ideas was to have default global options map, could be read from EDN-file, set like the registry etc and which could be tune all different parts of the system. Not there yet.#2022-08-1513:37pinkfrogI’d like to learn more on the scale well part.
When I am writing something only used internally, I will use qualified keywords, because this helps jump to usage, bulk rename, etc.
But when I am providing some opts map for the user, I will use simple keyword. Mandating qualified keywords kinda complicates the opts map.#2022-08-1515:33ikitommiMalli options has :`registry` being the only reserved simple key. There are some qualified keys used in core like ::m/walk-entry-vals, which are relevant only in a special use case (here, using m/walk). If we had two option arguments ("how the system is configured" and "how this use case should work") we could use simple keys in both. Now, it's just one arg, and I think it's better to namespace the latter keys to avoid conflicts and to document which ns owns them. You can merge 'em all into one map and pass it everywhere.#2022-08-1515:35ikitommiI recall there are two open issues related to how to manage the global options better. Both would introduce a breaking change in the public APIs.#2022-08-1515:36ikitommiI would like to remove the "pass in optional options arg to every function", have ideas on that, but, need to think about it first.#2022-08-1515:36ikitommiHere is a good example on bad and good option naming: https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L332-L361#2022-08-1714:30afleckis there an “exclusive or” way of combining schemas? say I have
(def foo-schema [:map [:id string?] [:foo number?]])
(def bar-schema [:map [:id string?] [:bar number?]])
(def foobar-schema [:orn [:foo foo-schema] [:bar bar-schema]])
if I do
(me/humanize (m/explain foobar-schema {:id :k :foo 3}))
I get
{:id ["should be a string" "should be a string"],
:bar ["missing required key"]}
but ideally I only want to see a single :id ["should be a string"] error. I guess put more abstractly I’m wondering if there’s a way to know “minimal number of problems that if fixed would make the value valid”#2022-08-1714:43afleckhmm seeming like maybe a multi schema could potentially be helpful#2022-08-1816:47ikitommi:multi is good option here#2022-08-1719:52Colin P. HillIs this a bug, or am I misunderstanding something? :thinking_face: It looks like it's complaining about the :=> schema only having one child, when it clearly has two.
(defn test-fn
{:malli/schema [:=>
[:cat :int]
[:int]]}
[n]
n)
=> #'user/test-fn
(require '[malli.dev :as dev])
=> nil
(dev/start!)
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
:malli.core/child-error {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}#2022-08-1720:08Noah Bogartlooks like it's discarding :int , and then discarding the empty vector too? what happens if you change it to int??#2022-08-1720:08Colin P. HillIdentical error#2022-08-1813:55Noah Bogartsorry to not be more help, that's super weird#2022-08-1813:57Colin P. HillVersion 0.8.4, in case anyone wants to see if they can reproduce it#2022-08-1813:59Noah Bogartcan you share the full stack trace? maybe that will help#2022-08-1814:04Colin P. Hill#error {
:cause ":malli.core/child-error {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}"
:data {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}}
:via
[{:type clojure.lang.ExceptionInfo
:message ":malli.core/child-error {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}"
:data {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}}
:at [malli.core$_fail_BANG_ invokeStatic "core.cljc" 136]}]
:trace
[[malli.core$_fail_BANG_ invokeStatic "core.cljc" 136]
[malli.core$_fail_BANG_ invoke "core.cljc" 134]
[malli.core$_check_children_BANG_ invokeStatic "core.cljc" 162]
[malli.core$_check_children_BANG_ invoke "core.cljc" 154]
[malli.core$__EQ__GT$reify__7246 _into_schema "core.cljc" 1694]
[malli.core$into_schema invokeStatic "core.cljc" 1921]
[malli.core$into_schema invoke "core.cljc" 1912]
[malli.core$schema invokeStatic "core.cljc" 1982]
[malli.core$schema invoke "core.cljc" 1963]
[malli.core$function_schema invokeStatic "core.cljc" 2390]
[malli.core$function_schema invoke "core.cljc" 2387]
[malli.core$function_schema invokeStatic "core.cljc" 2388]
[malli.core$function_schema invoke "core.cljc" 2387]
[malli.core$_register_function_schema_BANG_ invokeStatic "core.cljc" 2398]
[malli.core$_register_function_schema_BANG_ invoke "core.cljc" 2395]
[malli.core$_register_function_schema_BANG_ invokeStatic "core.cljc" 2396]
[malli.core$_register_function_schema_BANG_ invoke "core.cljc" 2395]
[malli.instrument$_collect_BANG_ invokeStatic "instrument.clj" 41]
[malli.instrument$_collect_BANG_ invoke "instrument.clj" 39]
[malli.instrument$collect_BANG_$fn__15014 invoke "instrument.clj" 70]
[clojure.core.protocols$iter_reduce invokeStatic "protocols.clj" 49]
[clojure.core.protocols$fn__8238 invokeStatic "protocols.clj" 75]
[clojure.core.protocols$fn__8238 invoke "protocols.clj" 75]
[clojure.core.protocols$fn__8178$G__8173__8191 invoke "protocols.clj" 13]
[clojure.core$reduce invokeStatic "core.clj" 6886]
[clojure.core$reduce invoke "core.clj" 6868]
[malli.instrument$collect_BANG_ invokeStatic "instrument.clj" 70]
[malli.instrument$collect_BANG_ invoke "instrument.clj" 58]
[malli.dev$start_BANG_ invokeStatic "dev.clj" 23]
[malli.dev$start_BANG_ invoke "dev.clj" 15]
[malli.dev$start_BANG_ invokeStatic "dev.clj" 20]
[malli.dev$start_BANG_ invoke "dev.clj" 15]
[user$eval15353 invokeStatic "scratch_16.clj" 2]
[user$eval15353 invoke "scratch_16.clj" 10]
[clojure.lang.Compiler eval "Compiler.java" 7194]
[clojure.lang.Compiler eval "Compiler.java" 7149]
[clojure.core$eval invokeStatic "core.clj" 3215]
[clojure.core$eval invoke "core.clj" 3211]
[clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
[clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
[clojure.main$repl$fn__9215 invoke "main.clj" 458]
[clojure.main$repl invokeStatic "main.clj" 458]
[clojure.main$repl doInvoke "main.clj" 368]
[clojure.lang.RestFn invoke "RestFn.java" 1523]
[nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 79]
[nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 55]
[nrepl.middleware.interruptible_eval$interruptible_eval$fn__10823$fn__10827 invoke "interruptible_eval.clj" 142]
[clojure.lang.AFn run "AFn.java" 22]
[nrepl.middleware.session$session_exec$main_loop__10925$fn__10929 invoke "session.clj" 171]
[nrepl.middleware.session$session_exec$main_loop__10925 invoke "session.clj" 170]
[clojure.lang.AFn run "AFn.java" 22]
[java.lang.Thread run "Thread.java" 831]]}#2022-08-1814:16Colin P. HillIssue disappears when I bump to version 0.8.9, which I don't see any reason not to do, so I suppose it's immaterial#2022-08-1814:17Colin P. HillThough I don't see anything about it in the change notes so maybe it was accidentally fixed and could still use a regression test, idk#2022-08-1814:32Noah Bogarti bet this is it: https://github.com/metosin/malli/pull/690#2022-08-1814:32Colin P. Hillseems likely#2022-08-1814:34Noah Bogartglad upgrading is the solution, that's a very easy solution lol#2022-08-1914:26ikitommiHi, is there an example cljs-project with shadow-cljs + malli.dev instrumentation enabled? Starting to port a large app from spec to malli, interested in good practices on how to do dev/prod separation, pretty printing errors in html (for react error boundaries on dev errors + enabling sci-powered code editing on the browser).#2022-08-1914:30ikitommiGoals:
• great DX
• clear separation of dev & prod validation
• tools for building dynamic schema systems
• small bundle size#2022-08-1914:31borkdudefor SCI-powered code editing I would take a look at https://nextjournal.github.io/clojure-mode/#2022-08-2113:12ikitommithat's great. Not a "dependency + one-liner to get an editor", but awesome in features etc.#2022-08-2113:14ikitommiBut, is there a simple (read-only) clojure code formatter for html? Malli error pretty printing is basically a sequence of text blocks and code blocks.#2022-08-2215:42dvingoI started playing around with a setup but it's closed source at the moment.
I'd like some more details on what is meant by bullet 2 - separation of dev and prod.
and for bullet 3 what is dynamic schema? constructing schema at runtime? based on data?
for point 4, the entry namespace in my project includes https://malli.dev which will increase bundle size. With shadow-cljs I think one way to solve this is with ns-aliases but I haven't confirmed this works. I fear you may have to use 2 different builds one for dev one for prod.
I'm not sure about great DX - I haven't used the cljs instrumentation on a large project, I think https://github.com/CrypticButter/snoop may offer better DX but has tradeoffs of its own - using new syntax for defn mainly.
The reason is that the intsrumentation macro produces large amounts of code (a set! call for every instrumented fn plus the code to filter every ns + fn in them) so on large codebases I can foresee this being prohibitive perf-wise. I've thought a little bit about a shadow-cljs server plugin/extension that would track which functions are instrumented and which are stale, which would have to happen per client (open browser window running the app). Then the macro would have access to this state to only output instrumentation code for stale functions. I'm not sure how feasible that is though, or how much time it would take, or how maintainable it would be.#2022-08-2215:43dvingo@U21QNFC5C I think has been using the cljs instrumentation - any insights?#2022-08-2215:47ingesolThis is funny, was about to post a couple of questions about friction with tooling 🙂#2022-08-2215:51ingesolMaybe a bit into the weeds, but there are 2 issues holding us back from a full rollout right now:
• If you remove :malli/schema metadata from a function, it remains instrumented
• We are relying on function identity, in a way that is not easily changed, and that is broken by the wrapping function added in -instrument.
I think we have added the setup required, specifically:
• In shadow init, call md/start! before mounting our reagent comp
• In after-load, call md/start!
This is supposed to be sufficient according to the docs, but nothing is unstrumented until we call md/stop!#2022-08-2216:22ingesolAlso, all those ..instrumented ns.here is not very nice when there are tens and hundreds of them. Should ideally be grouped or optional.#2022-08-2217:10ingesolhttps://github.com/metosin/malli/issues/744#2022-09-1516:57ingesolAnother issue posted on CLJS instrumentation https://github.com/metosin/malli/issues/752#2022-08-2000:25geraldodevI have a react form that has a input :type "date" and the stored value is a string "2022-08-01". How to check for valid date in malli with clojurescript ?#2022-08-2013:57pppaulanyone here had success using malli.swagger?#2022-08-2014:05pppaul{:info {:title "Site Map"},
:paths {"" {:get {:parameters {:query {:type "object",
:properties {:email {:type "string"}}},
:path {:type "object",
:properties {:uuid {:type "string",
:format "uuid"}},
:required [:uuid]},
:formData {:type "object",
:properties {}}},
:responses {403 {:description "User tried to access something their not allowed to",
:custom/template "mvp/error_pages/403.html"},
401 {:description "Bad Auth/Cookie gets a redirect to login"}},
:produces ("text/html")},
:post {:parameters {:query {:type "object",
:properties {:email {:type "string"}}},
:path {:type "object",
:properties {:uuid {:type "string",
:format "uuid"}},
:required [:uuid]},
:formData {:type "object",
:properties {:email {:type "string"},
:otp {:type "array",
:items {:type "integer",
:format "int64"}}},
:required [:email :otp]}},
:responses {403 {:description "User tried to access something their not allowed to",
:custom/template "mvp/error_pages/403.html"},
401 {:description "Bad Auth/Cookie gets a redirect to login"}},
:produces ("text/html"),
:consumes ("multipart/form-data")}},
"/login3{uuid}" {}},
:basePath "/swagger"}
this is what i'm giving it#2022-08-2014:06Noah BogartMaybe thread this discussion?#2022-08-2014:10pppauli'm using https://github.com/metosin/ring-swagger maybe this doesn't work anymore? (it's pretty old). i looked at some of the tests and my swagger objects look pretty similar to the ones in the tests. i get complaints from swagger-jsonfn saying it doesn't know how to convert my object into a swagger schema#2022-08-2014:11pppaullooking at swagger's docs https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.3.md#parameterObject i think malli.swagger is generating wrong data#2022-08-2014:13valtteriCurrently it produces OpenAPI 2.x https://github.com/metosin/malli#swagger2#2022-08-2014:19pppaulwell, malli.swagger output is throwing errors for the other library for swagger that metosin made. i don't know what to do#2022-08-2014:20pppaul{
"skipParam": {
"name": "skip",
"in": "query",
"description": "number of items to skip",
"required": true,
"type": "integer",
"format": "int32"
},
"limitParam": {
"name": "limit",
"in": "query",
"description": "max records to return",
"required": true,
"type": "integer",
"format": "int32"
}
}
it's not outputting this style of object#2022-08-2014:28pppauli guess i have to study the swagger docs and make my own transformer#2022-08-2014:34valtteriPlease file an issue and we will look into it. 🙂#2022-08-2014:41valtteriI'm not sure if Tommi has intended that malli.swagger should play along with ring-swagger#2022-08-2014:41pppaul😞#2022-08-2014:44valtteriI don't also understand why would both be needed. AFAIK they both generate swagger.json#2022-08-2014:44valtteriThat you can use to render swagger ui#2022-08-2014:47valtteriDifference being that ring-swagger can generate the json from plumatic schema / clojure.spec and malli.swagger does that for malli schemas#2022-08-2014:53valtteriUhhhmm.. Need to correct myself. Looks like malli.swagger produces "chunks" of swagger 2.0 compatible JSON, not the complete swagger doc.#2022-08-2014:53pppaulok, i did not know that#2022-08-2014:54pppaulit makes sense that malli swagger wouldn't produce the whole swagger doc, i just don't know what to do with it's output#2022-08-2014:54valtteriWhich kinda makes sense since Malli knows only about the shape of data. It doesn't know anything about route definitions and other stuff that goes into swagger spec#2022-08-2014:55valtteriAre you using reitit by any chance?#2022-08-2014:55pppaulyada#2022-08-2014:55valtteriBecause there's reitit.swagger#2022-08-2014:56pppauli was hacking on yada swagger to try to get it working with malli#2022-08-2014:56pppaulchanging over to reitit is a bigger challenge#2022-08-2014:57valtteriYeah, I see. I don't know how Yada swagger works but in reitit it works so that it constructs the full swagger.json from route data + schema definitions. For schemas it supports plumatic, clojure.spec and malli.. And when it's constructing the swagger, it checks which impl it is, generates the needed JSON chunks and places them into the swagger doc#2022-08-2014:58pppaulyeah, i used malli swagger to generate the params data, but it still wasn't enough#2022-08-2015:01valtteriYeah.. I'm trying to look if ring-swagger has extension points where you could jack in malli#2022-08-2015:10valtteriMhmhmh to my 👀 it looks like some serious hacking would be required#2022-08-2015:22pppauli think i just have to change the params code, i already have merged route + method params#2022-08-2015:22pppauli just don't know what to change the params to, seems like i need to build ref structures and flatten out nested maps or something#2022-08-2015:24pppauli poked at it for a few hours, and i think i rather just see about using reitit...#2022-08-2113:11ikitommihere too https://github.com/metosin/reitit/tree/master/examples/ring-malli-swagger#2022-08-2117:32pppaulthanks, I think that's exactly what I need to figure out how to fix yada#2022-08-2211:40eskosSwagger is specifically pre-OpenAPI, which does get confusing. The name change and a whole bunch of backwards breaking changes happened with the release of OpenAPI 3.0, so a good mental note is that you really shouldn’t expect one from another.
This isn’t necessarily very helpful to the issue at hand, but still an unfortunate detail one should be aware of.#2022-08-2116:18Elirazhello! I'm trying to work with on my project#2022-08-2116:19Elirazand am getting the following issue:
The required namespace "malli.dev" is not available, it was required by "main/root.cljs".
"malli/dev.clj" was found on the classpath. Maybe this library only supports CLJ?
#2022-08-2215:43dvingoyea what ingesol said sounds right, there's some notes here too:
https://github.com/metosin/malli/blob/master/docs/clojurescript-function-instrumentation.md#2022-08-2116:19Elirazanybody knows what the issue is?#2022-08-2116:46ingesol@eliraz.kedmi157 You need to use malli.dev.cljs for CLJS I think.#2022-08-2214:32plinslooks like malli does not accepts a set as a schema
what would be the equivalent of (spec/valid? #{:a :b} :a) ?#2022-08-2214:33Noah Bogartis this just :enum?#2022-08-2214:34Noah Bogartone of n possibilities?#2022-08-2214:34plinslooks like so, thanks#2022-08-2312:52Chris O’DonnellDoes anyone know the status of https://github.com/metosin/malli/pull/545 ? Running into the limitations of inst? at work, and it would be great to have a standard solution.#2022-08-2814:30ikitommihighly needed, pushed up on the TODO to check it out.#2022-08-2814:54Chris O’DonnellThanks! Let me know if there's anything I can do to help.#2022-08-2613:49Setzer22Hi! We've just noticed that schemas using a registry don't work with methods from malli utils, is this intended?
This works as expected:
(m/entries [:map [:hola :string]]) ; => ([:hola [:malli.core/val :string]])
But this fails (returns nil, but the schema is the same)
(m/entries [:schema
{:registry {"Foo" [:map [:hola :string]]}}
"Foo"]) ; => nil#2022-08-2614:19Noah BogartYou need m/deref-all :
(m/entries
(m/deref-all
[:schema {:registry {"Foo" [:map [:hola :string]]}}
"Foo"]))
;=> ([:hola [:malli.core/val :string]])#2022-08-2614:24Setzer22awesome, thanks! 👍#2022-09-0112:39souenzzoHello
In the context of this issue: https://github.com/metosin/malli/issues/739#issuecomment-1232938081
Should :my/int represent something different from [:my/int] ? or they both different notations with the same meaning?
This behavior can be considered a bug?
(m/ast :my/int opts) #_#_=> {:type :malli.core/schema, :value :my/int}
(m/ast [:my/int] opts) #_#_=> {:type :my/int}#2022-09-0112:43ikitommioh, doesn’t look right. works the same, but extra wrapping :thinking_face:#2022-09-0112:44ikitommineed to think about the issue, thanks for a gentle reminder 🙂#2022-09-0112:45souenzzothe extra wrapping makes one turn into $ref and another turn into a simple type, in json-schema generation.#2022-09-0112:46souenzzoI'm unsure if I should consider it a json-schema bug, or a malli bug.#2022-09-0122:44Justin ReedHow do I modify this:
[:map
[:key-0 uuid?]
[:key-1 string?]
[:key-2 [:string {:max 9 :min 9}]]
such that a map must have one of these three keys? For example,
{:key-0 (UUID/randomUUID)}
{:key-1 "HELLO WORLD"}
{:key-2 "123456789"}
{:key-1 "HELLO WORLD" :key-2 "123456789"}
are all valid, but
{}
{:key-4 "SOME OTHER KEY"}
are not.
I didn't find a direct way of doing this in the docs, so I thought I'd ask before I ventured down the path of cobbling it together with :and, :or, etc.#2022-09-0210:04iarenazaWhile it's not explicitly described as such, there is a hint about that possibility in the example shown in the paragraph with the text "Finding all subschemas with paths, retaining order:" There, the :fn schema is used to check that either the :streeet or the :lonlat keys are present in the :map schema (using the :and schema to tie both conditions). So something like this should work:
(mapv (fn [m]
(malli/validate [:and
[:map
[:key-0 {:optional true} uuid?]
[:key-1 {:optional true} string?]
[:key-2 {:optional true} [:string {:max 9 :min 9}]]]
[:fn (fn [{:keys [key-0 key-1 key-2]}]
(or key-0 key-1 key-2))]]
m))
[{:key-0 (UUID/randomUUID)}
{:key-1 "HELLO WORLD"}
{:key-2 "123456789"}
{:key-1 "HELLO WORLD" :key-2 "123456789"}
{}
{:key-4 "SOME OTHER KEY"}])#2022-09-0215:35emccue[:or
[:map {:closed true}
[:key-0 uuid?]]
[:map {:closed true}
[:key-1 string?]]
[:map {:closed true}
[:key-2 [:string {:max 9 :min 9}]]
[:map {:closed true}
[:key-0 uuid?]
[:key-1 string?]]
[:map {:closed true}
[:key-1 string?]
[:key-2 [:string {:max 9 :min 9}]]]
[:map {:closed true}
[:key-0 uuid?]
[:key-2 [:string {:max 9 :min 9}]]]
[:map {:closed true}
[:key-0 uuid?]
[:key-1 string?]
[:key-2 [:string {:max 9 :min 9}]]]
#2022-09-0215:38emccueis my first go at it - obviously you'd write something that would produce that#2022-09-0217:32Justin ReedThank you both for your help. I was about to go down the road @U3JH98J4R demonstrated, but was hoping for something less redundant. @U8T05KBEW’s nudge is exactly what I was looking for.#2022-09-0403:35Jungwoo KimHi, is there any way to check var?
(m/validate [:map
[:my-fn symbol?]] {:my-fn 'clojure.core/identity}) ;; obviously true
(m/validate [:map
[:my-fn var?]] {:my-fn (requiring-resolve 'clojure.core/identity)}) ;; expected true but invalid schema
As docs, https://github.com/metosin/malli#mallicorepredicate-schemas doesn’t support var? . How do I validate var? ?#2022-09-0410:41respatialized[:my-fn [:fn var?]] If nothing else you could use a predicate schema#2022-09-0423:33Jungwoo KimAhh that’s so useful ! thank you!#2022-09-0807:04Ferdinand BeyerIt seems that I can't use recursive schemas in regular expression schemas?
(def selector-schema
(malli/schema
[:schema {:registry {::selector [:and vector? [:+ [:cat keyword? [:? [:ref ::selector]]]]]}}
[:ref ::selector]]))
(malli/validate selector-schema [:foo [:bar]])
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/potentially-recursive-seqex [:ref :user/selector]
Is there another way to specify this? I basically want to specify a vector of keywords, that are optionally followed by a vector that conforms to the same spec...#2022-09-0807:09ikitommiyou can’t inline the recursive references (reasoning https://github.com/metosin/malli/blob/master/src/malli/impl/regex.cljc), but you can use them, with explicit :schema wrapping, e.g.
(def selector-schema
(malli/schema
[:schema {:registry {::selector [:and vector? [:+ [:cat keyword? [:? [:schema [:ref ::selector]]]]]]}}
[:ref ::selector]]))#2022-09-0807:10Ferdinand BeyerThanks a bunch!#2022-09-0807:10Ferdinand BeyerThis works!#2022-09-0807:10ikitommithere was a discussion of auto-wrapping :refs here, but it might be more confusing.#2022-09-0807:10ikitommiYour welcome 🙂#2022-09-0807:11Ferdinand BeyerI wonder if we should place a hint to that somewhere in the README. Happy to give it a try and PR#2022-09-0807:11Ferdinand BeyerI was about to give up already, would never have guessed to wrap it in [:schema]#2022-09-0807:11ikitommi> As all these examples show, the “seqex” operators take any non-seqex child schema to mean a sequence of one element that matches that schema. To force that behaviour for a seqex child :schema can be used:
> (m/validate
> [:cat [:= :names] [:schema [:* string?]] [:= :nums] [:schema [:* number?]]]
> [:names ["a" "b"] :nums [1 2 3]])
> ; => true
>
> ;; whereas
> (m/validate
> [:cat [:= :names] [:* string?] [:= :nums] [:* number?]]
> [:names "a" "b" :nums 1 2 3])
> ; => true
> #2022-09-0807:12ikitommithat’s on the README, which is bit overgrown…#2022-09-0807:12Ferdinand BeyerOh man.#2022-09-0807:12Ferdinand BeyerSorry 🙂#2022-09-0807:12ikitomminp#2022-09-0807:13Ferdinand BeyerNote to self: Always read the whole section#2022-09-1210:34CarloCan I generate a value from a registry? Something like:
(def reg
{::id int?
::name string?
::user [:tuple ::id ::name]})
(mg/generate {:registry reg} ::user)#2022-09-1213:38Ferdinand BeyerYes, by wrapping it in [:schema]:
(mg/generate [:schema {:registry reg} ::user])#2022-09-1213:54CarloThank you, could you expand a bit more on why this works/where to find docs?#2022-09-1217:40Ferdinand BeyerIt works since generate requires a schema to generate examples for. A registry is not a schema, just a map. So you need to create a schema that uses your registry.
As https://github.com/metosin/malli#local-registry, any schema can have a local registry property, e.g. [:map {:registry ,,,}]. Since you don't have such a schema in your case, you can use the generic [:schema] one, and just refer to a named one from the local registry.
An alternative would be the m/schema function to instantiate the schema explicitly. Remember that forms such as [:map] are not schemas yet, but will be instantiated on demand. You can explicitly turn them into schemas using m/schema.
Hope that helps!#2022-09-1314:10CarloThat is much more clear, thanks for the thoughtful answer @U031CHTGX1T#2022-09-1307:04Martynas MIs there a decoder for malli schema itself from JSON?
For instance if I have this schema:
[:map [:key {:optional true} :string]]
Can I parse it from this JSON?
["map",
["key", { "optional": true }, "string"]
]#2022-09-1310:04ikitommi[:map ["key" {:optional true} :string]] is also a valid schema, so one can’t guess from the JSON example that the "key" should be handled as a keyword.#2022-09-1310:05ikitommibut, if someone would write schemas of all malli schema syntaxes, one could do everything else (but not guess the string->kw things)#2022-09-1310:13Martynas MOk. That means that a dumbed-down version would work :thinking_face:#2022-09-1312:19ikitommiyes. If you want all maps to have keyword keys, you can:
1. read in from json
2. clojure-walk all vectors to have first arg as keyword
3. m/schema
4. m/walk and convert all map keys into keyword
5. profit#2022-09-1313:19Martynas MThere is also a difference between keyword and symbol .
And that's also significant because :inst? ,`inst?` and "inst?" are different :thinking_face:#2022-09-1315:12ikitommimaybe Tagged JSON? https://github.com/metosin/jsonista#tagged-json#2022-09-1315:15Martynas MI was thinking about dumbing down the schema and making everything a keyword or a number :thinking_face:
I don't need all power.#2022-09-1410:04juhoteperiIf you know all your map keys are going to be keywords, you could use walker and transform to change map keys to keywords from strings, in the read schema.#2022-09-1410:04juhoteperiOr you if just write limited Malli schema for the Malli schemas you need to support, you can use :keyword there to read the JSON strings as keys for the map keys.#2022-09-1410:39Martynas MI was thinking about making a subset of malli by doing something similar to a custom parser.
And then use :inst instead of inst? and :number instead of number? to make it all very uniform for the reader and parser. And after parsing I would do clojure.walk/postwalk or something similar to replace into the functions that malli understands.
So I should probably implement parsing for all basic types, :map , :set and some basic options like :optional. I think I don't need seq regex matching in this specific use as this will only be used to define a data structure and sequences, maybe lists with order too but there shouldn't be tuples.#2022-09-1314:16Anders EknertHey! Just discovered malli, and it looks like it does a lot of the things I need it for 😃 Could be I’ve missed something in the README, but one thing that isn’t immediately apparent to me, is if/how I can do validation of values that reference other attributes? So say that I have a map like:
{
"min": 10,
"max": 20
}
and I want to ensure that the min value is never greater than the value for max , etc#2022-09-1314:58lreadThe malli playground is a great place to discover malli features.
Maybe https://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D will give you inspiration?#2022-09-1315:00Anders EknertOoohh, that's very nice! Thanks very much 😃#2022-09-1315:06lreadYou are most welcome! I find it a great place to explore.#2022-09-1315:10Anders EknertFor sure! For whatever reason, I hadn't registered that was a thing before. Much appreciated 👍#2022-09-1409:01Anders EknertBack again 😅 While the custom validator function works for validation, I’ve found it breaks the default value transformer. My code looks something like this:
(def Conf
[:map
[:polling
[:and
[:map {:default {}}
[:min-delay-seconds [int? {:default 60}]]
[:max-delay-seconds [int? {:default 120}]]]
[:fn
{:error/message "max polling delay must be >= min polling delay"}
(fn [{:keys [min-delay-seconds max-delay-seconds]}]
(> min-delay-seconds max-delay-seconds))]]]])
Without the :and + :fn :
(m/decode Conf {} mt/default-value-transformer)
=> {:polling {:min-delay-seconds 60, :max-delay-seconds 120}}
And once that’s been added, the map comes back empty:
(m/decode Conf {} mt/default-value-transformer)
=> {}
I assume I’m doing something wrong, but I struggle with understanding what 🙂#2022-09-1414:12lreadNot at my dev box right now, but maybe due to {:default {}}?#2022-09-1414:17Anders EknertDoesn’t make a difference I’m afraid… I’ve tried a large number of combinations, but apparently not the right one 😅#2022-09-1414:47lreadI don’t want to to think I know malli well at all, so there might be other ways.
Moving the :default {} up seems to work:
(require '[malli.core :as m]
'[malli.transform :as mt])
(def Conf
[:map
[:polling {:default {}}
[:and
[:map
[:min-delay-seconds [int? {:default 60}]]
[:max-delay-seconds [int? {:default 120}]]]
[:fn
{:error/message "max polling delay must be >= min polling delay"}
(fn [{:keys [min-delay-seconds max-delay-seconds]}]
(> min-delay-seconds max-delay-seconds))]]]])
(m/decode Conf {} mt/default-value-transformer)
;; => {:polling {:min-delay-seconds 60, :max-delay-seconds 120}}
(m/decode Conf {:polling {:min-delay-seconds 1231}} mt/default-value-transformer)
;; => {:polling {:min-delay-seconds 1231, :max-delay-seconds 120}}
#2022-09-1414:50Anders EknertWow, it does! I must have tried pretty much any combination but that one facepalm Thanks again, Lee! ⭐️#2022-09-1414:51lreadMy pleasure, more experienced mallites might chime in, but the above looks ok to me.#2022-09-1317:37marciolHey, I have some code that have dependencies on spec, something that I cannot address in the near term, so I still need to duplicate malli and spec schemas everywhere. I'm just wondering if there is a lib that allows some sort of translation of malli schemas into spec schemas, in a way that we can make other spec dependent code thinks that a malli schema is a spec schema. Does anyone knows about some think like this?#2022-09-1318:31pithylessI'm aware of: https://github.com/dvingo/malli-code-gen/blob/main/thoughts.md#clojurespecalpha
https://github.com/dvingo/malli-code-gen/blob/main/src/main/space/matterandvoid/malli_gen/clojure_alpha_specs.cljc
I remember reading the docs, but have no idea how complete the actual implementation is.#2022-09-1318:32marciolI will track this, can be a point where I can start. Thanks#2022-09-1318:38pithylessI also remember this presentation showing how to generate domain models from malli (similar to the above repo thoughts). It's not directly related to clojure.spec, but perhaps you may gleam some insights if you go down the road of actually implementing this kind of generator.
https://www.youtube.com/watch?v=ww9yR_rbgQs#2022-09-1318:42pithylessI think both of these things came about before malli introduced https://github.com/metosin/malli#qualified-keys-in-a-map - perhaps with some clever use of macros and custom malli registries, you can get 80% there without much effort?#2022-09-1318:43marciolLets see, there is a lot of duplication going on so I need to solve it through some clever strategy#2022-09-1319:31dvingoYea, I paused working on that, but the good news is that since then malli added malli.core/ast which makes it a lot easier to parse and walk schemas.
I have a transform for taking malli schemas describing a hashmap and producing pathom output vectors that is working:
https://gist.github.com/dvingo/213633acfdd520bddcdc91fc1c7b9e44
you should be able to do something similar to output clojure specs. The Kondo output has a more complete set of schema types:
https://github.com/metosin/malli/blob/master/src/malli/clj_kondo.cljc#2022-09-1319:32marciol🙌#2022-09-1319:33dvingoI think @U0A5V8ZR6 continued working on something similar
https://github.com/Blasterai/malli-datomic/blob/master/src/blasterai/malli_datomic/spec_utils.cljc#2022-09-1321:28escherizeI want to use malli for the clj-kondo type-mismatch stuff on a large clojure project. But I’d rather not add it as a dependency. Is there a way to make malli.core/=> annotate functions in other namespaces? Then I can put all my m/=>’s into an un-committed namespace and still get the benefits of (). I looked at the code, and adding the feature doesn’t seem too tough. But I may be missing another approach#2022-09-1322:30dvingoyea that should be possible with a little custom code. Underneath => just calls -register-function-schema
https://github.com/metosin/malli/blob/bf92680ad76e57697261dd45c52dd31ffa9a8e1e/src/malli/instrument.clj#L41
which takes an ns symbol, a name symbol, a schema and a metadata map#2022-09-1406:12ikitommimaybe m/=> should allow qualified symbols too? e.g.
(m/=> my.domain/foo ...)
#2022-09-1406:12ikitommiwould not be a big change I guess: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2440-L2442#2022-09-1321:52escherizeHow do I make a schema for a vector that has 3 items: int, nil, string? e.g. [1 nil "a"] I can make a vector that is homogenous, or I can make a list using sequence schema. But I can’t figure this one out#2022-09-1404:22ikitommitry [:tuple :int :nil :string]#2022-09-1417:04escherizeExactly right! In hindsight I could have also:
(mp/provide
[[1 nil "x"]
[2 nil "y"]
[1 nil "z"]]
{:malli.provider/tuple-threshold 3})#2022-09-1409:01Anders EknertBack again 😅 While the custom validator function works for validation, I’ve found it breaks the default value transformer. My code looks something like this:
(def Conf
[:map
[:polling
[:and
[:map {:default {}}
[:min-delay-seconds [int? {:default 60}]]
[:max-delay-seconds [int? {:default 120}]]]
[:fn
{:error/message "max polling delay must be >= min polling delay"}
(fn [{:keys [min-delay-seconds max-delay-seconds]}]
(> min-delay-seconds max-delay-seconds))]]]])
Without the :and + :fn :
(m/decode Conf {} mt/default-value-transformer)
=> {:polling {:min-delay-seconds 60, :max-delay-seconds 120}}
And once that’s been added, the map comes back empty:
(m/decode Conf {} mt/default-value-transformer)
=> {}
I assume I’m doing something wrong, but I struggle with understanding what 🙂#2022-09-1605:39robert-stuttafordjust sanity-checking that this is appropriate - when using a registry, and having registered a spec for :my/keyword e.g. :string, then using that keyword in a :map, i simply do [:map [:my/keyword :my/keyword]] - there isn't some simpler syntax to use similar to clojure.spec.alpha/keys ?#2022-09-1613:52kennyI believe you can replace the vector with just the qualified keyword. e.g., [:map :my/keyword]#2022-09-1614:03robert-stuttafordsweet i'll try that!#2022-09-1605:40robert-stuttafordalso, i can use recursive :ref in a mutable registry, right? update: yes you can#2022-09-1619:06John MaruskaIs there a way to merge two constrained maps (`[:and [:map ...] [:fn ...]]`) together that doesn't drop the :fns of the second map? I haven't been able to figure it out so far, so (gen/sample-seq (mg/generator my-schema)) is giving results that fail m/validate#2022-09-1812:38ikitommi:thinking_face: with the latest master:
(mu/merge
[:and
[:map [:x :int]]
[:fn 'map?]]
[:and
[:map [:y :int]]
[:fn 'coll?]])
;[:and
; [:map [:x :int] [:y :int]]
; [:fn map?]
; [:fn coll?]]
#2022-09-1812:39ikitommilooks correct @U8VJYTQ76?#2022-09-1915:01John MaruskaGuess I just needed the weekend break -- found the issue this morning inside the function in my :fn#2022-09-1811:29Ory BandHi. I'm trying ot use a default fn value transformer from the tips section (https://github.com/metosin/malli/blob/master/docs/tips.md#default-value-from-a-function) and also discussed here https://clojurians.slack.com/archives/CLDK6MFMK/p1626171933375100,
but sci fails with "could not resolve symbol" with the function i'm using. Any idea why?
; (err) Execution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:39).
; (err) Could not resolve symbol: get-num-cpus
#2022-09-1811:40Ory Bandgot a response from borkdude here, unclear how to proceed though - how to pass a namespace? https://clojurians.slack.com/archives/C015LCR9MHD/p1663501053285409#2022-09-1812:40ikitommiyou can pass in the normal sci-options via :malli.core/sci-options. what ever sci accepts, is accepted here too:
(m/validate
[:fn '(fn [arg] (malli.impl.util/-invalid? arg))]
::m/invalid
{::m/sci-options {:namespaces {'malli.impl.util {'-invalid? malli.impl.util/-invalid?}}}})
; => true#2022-09-1812:41ikitommisci readme seems to have tips and helpers on how to bind multiple things at once.#2022-09-1812:41Ory Bandspecifically in my case, how can i add my custom fn get-num-cpu?#2022-09-1812:44ikitommi(defn get-num-cpus []
(-> (Runtime/getRuntime) .availableProcessors))
(m/validate
[:fn '(fn [arg] (= arg (get-num-cpus)))]
10
{::m/sci-options {:namespaces {'user {'get-num-cpus get-num-cpus}}}})
; => true#2022-09-1812:47ikitommidoes that solve the issue?#2022-09-1812:47ikitommibut, do you need sci here?#2022-09-1812:47Ory Bandthanks tommi! testing, getting back to you soon#2022-09-1812:47Ory Bandhmm, i thought i did. don't i? i need to calculate the value based on amount of available cpus#2022-09-1812:47ikitommiyou could also just remove the ' and say #(hash-map :thread-count (* 5 (get-num-cpus)))#2022-09-1812:48Ory Bandwait. i could do that? i thought you couldn't pass functions#2022-09-1812:48ikitommim/eval takes any function or a source code. for the latter, it uses sci to interpret it.#2022-09-1812:48ikitommiyou can’t store normal functions as data (over wire / db etc), but if you don’t need that, you don’t have to use sci.#2022-09-1812:49Ory Bandand i can put a fn under the :default key and it would work?#2022-09-1812:50ikitommiyes#2022-09-1812:50ikitommi(m/validate
[:fn #(= % (get-num-cpus))]
10)
; => true
#2022-09-1812:50Ory Bandamazing. i'm going to test that and get back to you.
another question: what if i need to calculate the :default based on another key's input?#2022-09-1812:50Ory Bandnot storing this as data#2022-09-1812:51Ory Bandsee this example, how can i avoid sci?
[:parallel-pull-count
{:title "Pull messages thread count"
:description ""
:default 1}
pos-int?]
[:ack-and-lease-extension-thread-count
{:title "ACK processing and lease extension thread count"
:description ""
; immitate java-pubsub's default value
;
:default-fn '#(max 6 (* 2 (:parallel-pull-count %)))}
pos-int?]]]])#2022-09-1812:53ikitommicurrently, the transformation interceptors don’t see their parents, so, attaching a transformer into a map key -> just sees that key & it’s children. This could be changed, but would need an issue and some thinking on how to do that.#2022-09-1812:54ikitommibut, you can attach the default values into the :map itself -> in that case the default-fn sees the whole map.#2022-09-1812:54Ory Bandyeah but in case the map was given by the user and one of the keys is missing, i want to support defaulting there as well#2022-09-1812:56Ory Bandbtw, just tried your suggestion and it doesn't work:
:default #(hash-map :thread-count (* 5 (get-num-cpus)))}
i'm getting
; (out) clojure.lang.ExceptionInfo: Invalid configuration {"parsing_errors" {:concurrency ["invalid type"]}}#2022-09-1813:04Ory Bandwhat am i doing wrong?#2022-09-1813:08ikitommithe :default key is defined in the default-value-transformer and it should be a value, e.g. (hash-map :thread-count (* 5 (get-num-cpus))). the default-fn-value-transformer defines a key :default-fn which can be a function, e.g. #(hash-map :thread-count (* 5 (get-num-cpus))).#2022-09-1813:10ikitommi(m/decode
[:map {:default {}}
[:x {:default (get-num-cpus)} string?]
[:y {:default-fn #(* (get-num-cpus) (:x %))} string?]]
nil
(mt/transformer
(mt/default-value-transformer)
(default-fn-value-transformer)))
; => {:x 10, :y 100}#2022-09-1813:10Ory Bandso i should use default-fn then#2022-09-1813:10ikitommiI think so#2022-09-1813:10Ory Bandtesting#2022-09-1813:40Ory Bandit works. thank you!#2022-09-1819:50Anders Eknert> currently, the transformation interceptors don’t see their parents, so, attaching a transformer into a map key -> just sees that key & it’s children. This could be changed, but would need an issue and some thinking on how to do that.
Interesting. That’s just the issue I was struggling with 😅 Should I create an issue?#2022-09-1820:03Anders EknertTo be more specific, I’m trying to get the name in the key of a :map-of construct, and use it as a default for one of the values in a sub-map#2022-09-2009:02hmadelaineHi everyone !
I have a question regarding malli.error/humanizeand a sequential schema.
I have modified the example given in the README.
Instead of having a nil latitude, I put a string.
(-> Address
(m/explain
{:id "ID676"
:tags #{:artesan :coffee :garden}
:address {:street "Ahlmanintie 29"
:city "paris"
:zip 33100
:lonlat [61.4858322, "23.89"]}})
(me/humanize))
=> {:address {:lonlat [nil ["should be a double"]]}}
I was expecting in the humanized error, this output :
{:address {:lonlat ["23.89" ["should be a double"]]}}
The result of m/explain indicates clearly in the errors that the input “23.89" is incorrect.
{:schema [:map
[:id string?]
[:tags [:set keyword?]]
[:address [:map [:street string?] [:city string?] [:zip int?] [:lonlat [:tuple double? double?]]]]],
:value {:id "Hiram",
:tags #{:coffee :artesan :garden},
:address {:street "Ahlmanintie 29", :city "paris", :zip 33100, :lonlat [61.4858322 "23.89"]}},
:errors ({:path [:address :lonlat 1], :in [:address :lonlat 1], :schema double?, :value "23.89"})}
Is it normal ?
Thnak you very much#2022-09-2014:00ikitommithis is normal, for sequential, nil are used to mask the valid values, e.g. “second and fourth elements in error” = [nil nil ["error"] nil ["error"]]#2022-09-2014:00ikitommiyou can configure how to show the valid values, e.g.
(-> Address
(m/explain
{:id "Lillan"
:tags #{:artesan "coffee" :garden "ground"}
:address {:street "Ahlmanintie 29"
:zip 33100
:lonlat [61.4858322, "23.7832851,17"]}})
(me/error-value {::me/mask-valid-values '...}))
;{:id ...
; :tags #{"coffee" "ground" ...}
; :address {:street ...
; :zip ...
; :lonlat [... "23.7832851,17"]}}#2022-09-2014:22hmadelaineAlright ! Thank you very much 👍#2022-09-2014:51hmadelaineIt seems that me/error-value is not available in 0.8.9 ?#2022-09-2015:07hmadelaineDo you plan to release malli this month ?#2022-09-2015:47ikitommiyrs#2022-09-2015:48hmadelaineThank you for the excellent work 🙏#2022-09-2108:53hmadelaineHi !
With metosin/spec-tools I use the function (select-spec spec value)
I did not find an equivalent in malli
Do you plan to add it later or should I add it to my own ns malli.utils?
Thanks#2022-09-2109:15ikitommithere is no helper, but:
(m/decode schema value (mt/strip-extra-keys-transformer))
#2022-09-2109:16hmadelaineYes that is what I did 😉
Thank you#2022-09-2109:16ikitommifor perf:
(def select-address (m/decoder Address (mt/strip-extra-keys-transformer)))
(select-address value)
#2022-09-2109:16ikitommiif you have both spec-tools and malli, please share perf numbers on that. I think it’s… big#2022-09-2109:17hmadelaineGood idea#2022-09-2216:57Ben SlessFound some performance improvement opportunities around parsing, especially around regex parsers.
Also, trying to fold and/or parsers, but it's confusing 🙃#2022-09-2217:21ikitommilooking forward!#2022-09-2217:21ikitommito#2022-09-2217:25Ben SlessWould appreciate a pointer regarding what I'm getting wrong with orn
this is the or parser:
(defn- -or->parser
[children]
(fn [f]
(let [parsers (-vmap f children)]
#?(:clj
(reduce
(fn [acc parser]
(fn [data]
(let [parsed (acc data)]
(if (miu/-invalid? parsed)
(parser data)
parsed))))
(fn [_] ::invalid)
(reverse parsers))
:cljs
#(reduce (fn [_ parser] (miu/-map-valid reduced (parser %))) ::invalid parsers)))))
And the orn parser doesn't work
(defn- -orn->parser
[this]
#?(:clj
(reduce
(fn [acc [k _ c]]
(let [parser (-parser c)]
(fn [data]
(let [parsed (acc data)]
(if (miu/-invalid? parsed)
(parser data)
(miu/-tagged k parsed))))))
(fn [_] ::invalid)
(reverse (-children this)))
:cljs
(let [parsers (-vmap (fn [[k _ c]]
(let [c (-parser c)]
(fn [x] (miu/-map-valid #(reduced (miu/-tagged k %)) (c x)))))
(-children this))]
(fn [x] (reduce (fn [_ parser] (parser x)) x parsers)))))
#2022-09-2406:07Ben SlessAnyway, first MR which I'm sure of submitted. Would you like to see before/after comparisons?#2022-09-2217:45Ben SlessI decided to break things apart and start with the improvements I'm sure of
https://github.com/metosin/malli/pull/756#2022-09-2217:47Ben SlessOne radical thing I've considered trying is pivoting the values array in the Cache to as many arrays as CacheEntry has fields, avoids allocation and makes the entire thing cache friendly#2022-09-2218:02dvingoafter a comment by Thomas Heller (https://github.com/metosin/malli/pull/754#issuecomment-1252581659) I refactored the cljs function instrumentation https://github.com/metosin/malli/pull/755 to happen all in cljs with only collecting schemas happening in a macro. The benefits include the namespaces matching with the clojure versions (aside from malli.dev.cljs), a simpler mental model as instrumentation doesn't occur via generated code, and a more straight forward implementation that matches clojure's pretty closely.#2022-09-2220:17kennyWhen working with Malli and generators, how do folks typically handle lazy loading of test.check?#2022-09-2221:51dvingohttps://github.com/borkdude/dynaload is great for this sort of thing#2022-09-2301:08kennyCool lib :thumbsup::skin-tone-2: How do you set the generator on a schema to be dynamically loaded? i.e., if I'm setting :gen/gen key, it'd need to be loaded when setting it.#2022-09-2312:43dvingoi haven't done it myself, but perhaps walking the schema dynamically to produce a new schema?
https://github.com/metosin/malli#programming-with-schemas
the "Adding generated example values to Schemas" section in particular#2022-09-2409:51Ben Sless#2022-10-0707:54ikitommithis is awesome!!#2022-10-0708:37Ben SlessThank you!
It's also a really good benchmark case for parsing, which is what led to my other adventures 🙂#2022-09-2511:06delaguardoHi! Is there a way to make a schema to validate a map has a set of keys and validate the rest of the keys against :map-of schema? This is very close to additionalProperties directive in json-schema.#2022-09-2511:45delaguardojust found this - https://github.com/metosin/malli/issues/43 discussion#2022-09-2511:10delaguardo[:map {:extra-keys [:map-of {:min 1 :max 2} :string :string]}
[:x [:= "X"]]
[:y {:optional true} :string]]
I was thinking to add an option into -map-schema but I bit lost in the source code. Would appreciate any help to better understand how IntoSchema work internally 🙂 can’t find any low-level article about that#2022-10-0707:53ikitommithere is a old issue about this. is important.#2022-10-0708:30delaguardoSo far I made a custom -map-schema with :extra-entries option where it expects to have a schema to validate the “rest” of the map
(defn -map-schema
([]
(-map-schema {:naked-keys true}))
([opts] ;; :naked-keys, :lazy
^{:type ::m/into-schema}
(reify
m/AST
(-from-ast [parent ast options] (m/-from-entry-ast parent ast options))
m/IntoSchema
(-type [_] :map)
(-type-properties [_])
(-properties-schema [_ _])
(-children-schema [_ _])
(-into-schema [parent {:keys [closed extra-entries]
:or {extra-entries true}
:as properties} children options]
(let [entry-parser (m/-create-entry-parser children opts options)
form (delay (m/-create-entry-form parent properties entry-parser options))
cache (m/-create-cache options)
extra-entries (if closed false extra-entries)
extra-entries (cond (boolean? extra-entries)
(m/into-schema (if extra-entries :any 'empty?) nil nil options)
(vector? extra-entries)
(m/schema extra-entries options))
->parser (fn [this f]
(let [keyset (m/-entry-keyset (m/-entry-parser this))
parsers (cond->> (m/-vmap
(fn [[key {:keys [optional]} schema]]
(let [parser (f schema)]
(fn [m]
(if-let [e (find m key)]
(let [v (val e)
v* (parser v)]
(cond (miu/-invalid? v*) (reduced v*)
(identical? v* v) m
:else (assoc m key v*)))
(if optional m (reduced ::m/invalid))))))
(m/-children this))
extra-entries (cons (let [parser (f extra-entries)]
(fn [m]
(let [ks (keys keyset)
m' (select-keys m ks)
m (apply dissoc m ks)
m* (parser m)]
(cond (miu/-invalid? m*) (reduced m*)
:else (merge m' m*)))))))]
(fn [x] (if (map? x) (reduce (fn [m parser] (parser m)) x parsers) ::m/invalid))))]
^{:type ::m/schema}
(reify
m/AST
(-to-ast [this _] (m/-entry-ast this (m/-entry-keyset entry-parser)))
m/Schema
(-validator [this]
(let [keyset (m/-entry-keyset (m/-entry-parser this))
validators (cond-> (m/-vmap
(fn [[key {:keys [optional]} value]]
(let [valid? (m/-validator value)
default (boolean optional)]
(fn [^Associative m] (if-let [map-entry (.entryAt m key)] (valid? (.val map-entry)) default))))
(m/-children this))
extra-entries (conj (let [valid? (m/-validator extra-entries)]
(fn [^Associative m] (valid? (apply dissoc m (keys keyset)))))))
validate (miu/-every-pred validators)]
(fn [m] (and (map? m) (validate m)))))
(-explainer [this path]
(let [keyset (m/-entry-keyset (m/-entry-parser this))
explainers (cond-> (m/-vmap
(fn [[key {:keys [optional]} schema]]
(let [explainer (m/-explainer schema (conj path key))]
(fn [x in acc]
(if-let [e (find x key)]
(explainer (val e) (conj in key) acc)
(if-not optional
(conj acc (miu/-error (conj path key) (conj in key) this nil ::m/missing-key))
acc)))))
(m/-children this))
extra-entries (conj (let [explainer (m/-explainer extra-entries path)]
(fn [x in acc]
(explainer (apply dissoc x (keys keyset)) in acc)))))]
(fn [x in acc]
(if-not (map? x)
(conj acc (miu/-error path in this x ::m/invalid-type))
(reduce
(fn [acc explainer]
(explainer x in acc))
acc explainers)))))
(-parser [this] (->parser this m/-parser))
(-unparser [this] (->parser this m/-unparser))
(-transformer [this transformer method options]
(let [this-transformer (m/-value-transformer transformer this method options)
->children (reduce (fn [acc [k s]]
(let [t (m/-transformer s transformer method options)]
(cond-> acc t (conj [k t])))) [] (m/-entries this))
apply->children (when (seq ->children) (m/-map-transformer ->children))
apply->children (m/-guard map? apply->children)]
(m/-intercepting this-transformer apply->children)))
(-walk [this walker path options] (m/-walk-entries this walker path options))
(-properties [_] properties)
(-options [_] options)
(-children [_] (m/-entry-children entry-parser))
(-parent [_] parent)
(-form [_] @form)
m/EntrySchema
(-entries [_] (m/-entry-entries entry-parser))
(-entry-parser [_] entry-parser)
m/Cached
(-cache [_] cache)
m/LensSchema
(-keep [_] true)
(-get [this key default] (or (m/-get-entries this key default)
(and extra-entries
(m/-get extra-entries key default))))
(-set [this key value] (m/-set-entries this key value))))))))
#2022-10-0708:31delaguardoHowever it is incomplete and needs adjustments#2022-09-2615:12erwinrooijakkersIs there a way to describe that a map should contain either one specific key or another key?
e.g.
{:foo 1} ; correct
{:bar 1} ; correct
{} ; incorrect, should contain one of :foo or :bar#2022-09-2615:31delaguardo[:or [:map [:foo [:= 1]]] [:map [:bar [:= 1]]]]
#2022-09-2615:32erwinrooijakkersyep thanks, 🙂 but there are many more keys in the map so that would be loads of duplication, maybe there’s a way to do ir inside the map#2022-09-2615:38erwinrooijakkerse.g.,
user=> (m/validate [:map [:or [:foo :int]] [:bar :int]] {:quz 1})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
:malli.core/invalid-schema {:schema :x}#2022-09-2616:01hmadelaineYou could use a :fn schema for that purpose :
It is naive but it works
(def FooOrBar
[:and
[:map
[:foo {:optional true} number?]
[:bar {:optional true} number?]]
[:fn
{:error/message "Only one key must be present"}
(fn [data]
(= 1 (count data)))]])
(m/validate FooOrBar {:foo 1}) => true
(m/validate FooOrBar {:bar 1}) => true
(m/validate FooOrBar {:foo 1 :bar 1}) => false
(m/validate FooOrBar {}) => false#2022-09-2619:59pithyless> but there are many more keys in the map so that would be loads of duplication
The duplication can be resolved via composition, ala :merge , :union or even :select-keys.
Remember also, that you may need to close the map schemas appropriately, if you do not want both :foo and :bar to be valid together.
I would prefer :or over :and, since it is more likely to be generator friendly; but you may even consider a version using :multi with custom dispatcher that checks for the existence of certain keys.#2022-09-2914:26darnok@erwinrooijakkers Have you found a solution?#2022-09-2914:27darnokI have similar case, and I tried :and and :or and closing all maps. It doesn't work, because one of the specs won't pass the keys missing in another maps.#2022-09-2616:08robert-stuttafordhttps://twitter.com/RobStuttaford/status/1574430197079752706 (is it ok for me to paste a link to a tweet like this?)#2022-09-2620:16richiardiandreaThis is very nice, I wonder if we could do the same with cider...it would be very cool to be able to customize the jump to reference functionality#2022-09-2620:59DrLjótssonI am trying to get swagger-ui to work in reitit but am failing. I just want to double check if reitit + swagger-ui works when malli is used to spec the routes. The docs at https://github.com/metosin/reitit/blob/master/doc/ring/swagger.md say that "Reitit supports https://swagger.io/ documentation, thanks to https://github.com/metosin/schema-tools and https://github.com/metosin/spec-tools., i.e. malli is not mentioned.#2022-09-2707:02DrLjótssonI can confirm that swagger works with malli if reitit.coercion.malli/coercionis used.#2022-09-2809:21Martynas MWhat does this error mean?
https://github.com/metosin/malli/blob/c0965e2b2b37ea1d81e6f4ece7c87f56fa90398a/test/malli/core_test.cljc#L1799
I get it when I try to have a malli schema to decode malli tuples. And well... it has to be recursive :thinking_face:#2022-09-2809:23Martynas MI fixed the error by not using :ref in the definition like this:
::malli-tuple-schema [:cat
[:= :tuple]
[:* ::malli-schema]
#_[:* [:ref ::malli-schema]]]#2022-09-2813:38ambrosebsI think it means that any regex operation cannot contain a recursive schema. checking for :ref is a crude way of checking for recursion.#2022-09-2813:39ambrosebsI developed a more refined way to detect recursive specs in the generators namespace. perhaps using that here would be a nice enhancement.#2022-09-2813:41Martynas M> cannot contain a recursive schema
Well if I remove the :ref it will still reference the ::malli-schema but it will copy it. So the schema works the same way as it would work with :ref#2022-09-2813:42ambrosebsAgreed. That's the part that I think could potentially be improved.#2022-09-2813:43ambrosebseg., the ;; A bit undesirable, but intentional: test a few lines down from the test you linked.#2022-09-2813:43ambrosebsis ::malli-schema recursive?#2022-09-2813:43Martynas MEverything there is recursive 😄#2022-09-2813:43Martynas M::malli-schema [:or
[:ref ::malli-map-schema]
[:ref ::malli-set-schema]
[:ref ::malli-vector-schema]
[:ref ::malli-tuple-schema]
:keyword]#2022-09-2813:44ambrosebsah. I'm confused, you said you "fixed the error", what does that look like now?#2022-09-2813:44Martynas MThis looks like [:* ::malli-schema] and it works without the error#2022-09-2813:45ambrosebsthanks. now I understand how crude this check is 🙂#2022-09-2813:46ambrosebsI thought it was an overapproximation on checking for recursive specs. but it only guards against very specific recursive ones.#2022-09-2813:46ambrosebsunless I'm missing something.#2022-09-2813:47Martynas MThis is a more basic example that passes the checker:
[:schema
{:registry
{::malli-field-options [:map
[:optional {:optional true} :boolean]
[:min {:optional true} :int]
[:registry {:optional true} ::malli-map-schema]],
::malli-map-schema [:cat
[:= :map]
[:* [:or
[:tuple :keyword [:ref ::malli-schema]]
[:tuple :keyword [:ref ::malli-field-options] [:ref ::malli-schema]]]]],
::malli-schema [:or
[:ref ::malli-map-schema]
:keyword]}}
::malli-schema]#2022-09-2813:51ambrosebsok, and that one has no references/`:ref` directly on regex ops. can you post the full one that yields the error and requires a change?#2022-09-2813:55Martynas MI don't know. I can't find it#2022-09-2813:55Martynas MI'll probably be editing it more. Maybe I can find it later. But when I got that bug I was unsure what to do. So it's a tough one.#2022-09-2813:56ambrosebsnp, just sounds like a good test case. I get the gist of it.#2022-10-0707:52ikitommithe :ref can’t be used to expand/inline things into a sequential schema. you can use :refs, but need to wrap it into :schema - takes just one position in the sequence. This is an implementation decision, described in the ns. If there is a performant way not to do this, I’m all 👂s!#2022-09-2820:40DiegoGot a question about malli’s :re schemas.
I’ve got this code for validating email addresses:
(def email-address-regex #"(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])")
(def email-address? [:re {:error/message "Please provide a valid email address"} email-address-regex])
(me/humanize (m/explain email-address? "
It seems like :re is using re-find because re-find does return true, but what can I do to make :re behave more like re-match ?#2022-09-2909:02iarenazaYeah, it seems it's using re-find for validation: https://github.com/metosin/malli/blob/0.8.9/src/malli/core.cljc#L1350-L1351
In that case, you need to use the "beginning of line/string" (`^`) and "end of line/string" (`$`) anchors in your regular expression, so that re-find must match your regular expression over the whole input value, not just a part of it:
user> (require '[malli.core :as m])
nil
user> (require '[malli.error :as me])
nil
user> (def email-address-regex #"^(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:\
(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])$")
#'user/email-address-regex
user> (def email-address? [:re {:error/message "Please provide a valid email address"} email-address-regex])
#'user/email-address?
user> (me/humanize (m/explain email-address? "#2022-09-2919:54DiegoThanks!#2022-09-2919:05ingesolI tried to instrument a CLJS multimethod (the dispatch function), but it does not seem to work. I can call it with wrong input type, and it still works:
(defn my-dispatch
{:malli/schema [:=> [:cat :string] :keyword]}
[s]
(keyword s)
(defmulti a-multi-method
my-dispatch)
(defmethod a-multi-method :default [_]
:the-default)
;; The below does not crash like it should
(a-multi-method 4)
#2022-09-2920:28ambrosebsJust a hunch, but have you tried:
(defmulti a-multi-method
#(my-dispatch %))
#2022-09-3006:33ingesolThanks, that’s an interesting idea. The assumption is that the original function will be cached inside the multimethod storage? Anyway, it did unfortunately not work.#2022-09-3009:17ingesolAlso, to clarify, calling the my-dispatch function normally from somewhere else with wrong args triggers validation#2022-09-3013:30dvingoIt looks like this is due to how multimethods are implemented.
https://github.com/clojure/clojurescript/blob/961807166c8cf4d45a225d63416f06464fb27eaf/src/main/cljs/cljs/core.cljs#L11343
the dispatch-fn is passed into a JS object (deftype) and that captures the original function value. Just tried this in JS:
var afn = function afn(){ console.log('FIRST')}
y = {x: afn}
afn = function (){ console.log('SECOND')}
y.x()
// => FIRST
so I think this is what the problem is#2022-09-3013:32dvingowe may be able to mutate the MultiFn's dispatch-fn property when we instrument, but then the defmulti would have to have the instrumentation applied, not the function#2022-09-3014:36ambrosebs> Thanks, that’s an interesting idea. The assumption is that the original function will be cached inside the multimethod storage? Anyway, it did unfortunately not work.
Yes, I was also going to suggest #'my-dispatch but I forgot how vars work in CLJS.#2022-09-3016:40ingesolTaking one step back, the most valuable thing for our codebase would be to be able to declare the contract for the multimethod in general. Instrumenting the dispatch function is more like a hack to get at least some of the value. I know that instrumenting defmulti is probably a large and maybe not attractive subject, just mentioning it.#2022-09-3016:55ambrosebsAFAIK it is not possible in CLJ.#2022-09-3016:56ambrosebsprobably easier in cljs#2022-09-3017:05ingesolTo illustrate what I mean
(defmulti say-hi
{:malli/schema [:=> [:cat [:map [:nationality :keyword]]] :string]}
(fn [person]
(:nationality person)))
(defmethod say-hi :german
"Guten tag")#2022-09-3017:09ingesolI think I remember some earlier discussion indicating that the above is simplistic and probably not a good idea, but did not find it. And @U055XFK8V, just to clarify, this is the thing you are saying is not possible in CLJ?#2022-09-3017:47ambrosebsYes, I've looked at clojure.lang.MultiFn and there's no obvious way to achieve this.#2022-09-3017:47ambrosebsI managed to do this for defprotocol https://blog.ambrosebs.com/2022/09/08/schema-defprotocol.html#2022-09-3017:49ambrosebsactually, MultiFn isn't a final class so perhaps a wrapper can be created. I might look into that.#2022-09-3017:51ingesolBut that would require an alternative defmulti macro, right?#2022-09-3017:53ambrosebsI'm thinking we leave everything as-is, but an instrument! function when called on a multimethod would set the multimethod var to a thin wrapper around MultiFn.#2022-09-3017:53ambrosebsThe tricky part is making sure that wrapper works with existing defmethod calls. And if we subclass MultiFn, it might work.#2022-09-3017:54ingesolThis is all about the mechanics of doing the instrumentation, I also seem to remember there are issues with the semantics. Will try to look it up#2022-09-3018:00ingesolMaybe forget that last thing. So at least the way we use multimethods, it makes sense to enforce a shared input/output contract for all defmethods. The only thing that is not clear is how/where on the defmulti to define the schema.#2022-09-3018:00ingesolPinging @U055NJ5CC in case he has opinions on this#2022-09-3018:04ambrosebsThinking something like
public MyMultiFn extends MultiFn {
public MyMultiFn(IFn wrapper, MultiFn inner) { }
public Object invoke(args ...) {
wrapper.invoke(inner, args...);
}
....
}
(defn instrument-multi! [var checking-wrapper]
(alter-var-root var (fn [multi] (MyMultiFn. checking-wrapper multi)))
#2022-10-0200:23ambrosebsI managed to prototype defmulti instrumentation successfully for Prismatic Schema, so I take back my assertion that this was impossible.#2022-10-0707:44ikitommiWould be great if multimethods could be schematized. No opinions on the impl.#2022-10-0718:44ambrosebsOk I'll prep a proof of concept#2022-10-0113:09Chris O’DonnellPut up a small PR fixing a typo I found testing the json schema implementation: https://github.com/metosin/malli/pull/757. Haven't contributed to malli before; please let me know if there's anything else I should do.#2022-10-0312:25robert-stuttafordis there a Correct™ way to test if a value is a valid malli spec? i'd like to avoid try catch if that's possible#2022-10-0313:36Noah BogartYou mean “can the given form be used as a Malli schema”, not “does the given form conform to an existing malli schema”, right? #2022-10-0314:23respatializedAFAIK there isn't yet a self-describing meta-schema for malli schemas; the only way I've done this is indeed with try/catch on m/schema.#2022-10-0314:25robert-stuttafordthat's right @UEENNMX0T 🙂 thanks @UFTRLDZEW!#2022-10-0707:42ikitommiSchemas can define their schemas as schemas. The IntoSchema has the needed hooks, someone just needs to write the Schemas for properties & Children.#2022-10-0403:35dumratRan into a little perf trouble with malli sampling. Looks like performance is exponential wrt to sample size. Is there a reason for this?
Sample code:
(def cashflow-schema
(m/schema
[:map
[:type keyword?]
[:direction keyword?]
[:incurred-by {:optional true} [:map [:type keyword?] [:name string?]]]
[:amount int?]
[:categories [:set string?]]
;; [:date inst?]
[:description {:optional true} string?]
[:towards {:optional true} [:map [:type keyword?] [:name string?]]]]))
(defn get-sampling-data [sample-sizes]
(map (fn [ss] [ss ((comp first :mean)
(quick-benchmark (doseq [x (mg/sample cashflow-schema {:size ss})]
x) {}))])
sample-sizes))
(defn get-vis-data [sample-sizes]
(let [data (get-sampling-data sample-sizes)
values (map (fn [[ss time]] {:sample-size ss :time time}) data)]
{:data {:values values}
:encoding {:x {:field "sample-size" :type "quantitative"}
:y {:field "time" :type "quantitative"}}
:mark "line"}))
(get-vis-data (range 5 110 5))
(oz/view! *1)#2022-10-0403:50Ben SlessCan you def a generator from the schema and run again?#2022-10-0405:27dumratYes, that works! Looks like it's almost constant time now.#2022-10-0406:15Ben SlessAwesome!
All malli operations can instantiate single closures which already do all the heavy lifting and only need to validate, parse, w/e
You should also try a larger range, and add a doall around the generated result,it may be lazy#2022-10-0406:39dumratNow that you mention it, it might be lazy, I agree. Time doesn't look right.
But I tried generating samples with this and it's much faster than with previous method so for my purposes this is fine.
Let me check the perf anyway#2022-10-0406:44dumratOne question:
I used generator like this:
(defn generate-samples [sample-size]
(let [gen (mg/generator cashflow-schema)]
(doall (gen/sample gen sample-size))))
This is fine right?#2022-10-0407:12Ben SlessYes, but you can even def the generator right under the schema. As you can see it does not depend on the sample size at all#2022-10-0407:21dumratYeah, so doing this makes it significantly faster (For example, to generate 500 samples, my previous code takes 10s while generator takes 1.5s so there's a marked perf improvement. But the growth is still exponential. But I think I will leave it there. I originally wanted to generate around 1000 samples and it took too long that's why I delved into this.#2022-10-0410:00Ben SlessThe old nerd snipe 🙂#2022-10-0403:36dumrat#2022-10-0508:22DenisMcQuick question about malli with swagger. Right now, I define my reitit route definitions with malli, and malli-swagger generates the swagger definitions - all working well. However, recently I have added a couple of API responses where the malli definition is of the form [:or [<option1 payload>] [<option 2 payload>]] In this case, swagger seems to just report <option 1 payload> as the route payload. How would I go about updating swagger so that it displays the options to the API user? Thanks in advance.#2022-10-0512:32eskosMaybe https://github.com/metosin/malli#multi-schemas would do the trick?#2022-10-0722:38DenisMcI’ll give it a go 👍#2022-10-0520:06dangercoderDoes anyone know an idiomatic way of making malli always generate UTF8 strings? I currently use my own :utf8-string type instead of :string#2022-10-0520:14Noah BogartWhen is it not generating utf-8 strings?#2022-10-0520:23dangercoder(malli.generator/generate [:string {:min 10}])
#2022-10-0520:25dangercoderadding :min makes the generator not generate utf-8 strings consistently#2022-10-0611:56dangercoderhttps://github.com/metosin/malli/issues/758
Would be great if someone else could try it out and see if you get the same results#2022-10-0615:08ambrosebsI was the last person to touch that code so I should try it out...do you know how long this has happened and if there is a better generator to use?#2022-10-0615:41ambrosebslooks like this case is built on gen/char .#2022-10-0615:46ambrosebsperhaps gen/char-ascii or gen/char-alphanumeric should be used.#2022-10-0616:12ambrosebsproposed a fix https://github.com/metosin/malli/pull/759#2022-10-0817:33dangercoderThanks for the fix :heart_hands:#2022-10-0707:59ikitommihad a busy month and ending that now with a 1 week family vacation under a 🌴. Just kooked at the PRs, merged most, big thanks to all contributors! Will cut a release after a week.#2022-10-0710:36Noah BogartCongrats on the vacation! #2022-10-0708:18dharrigan🥳#2022-10-0819:45skynethow do I use the function generated from :malli/gen true in function metadata <https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schema-metadata>?
the doc says "Setting :malli/gen to true while function body generation is enabled with mi/instrument! allows body to be generated, to return valid generated data."
for example
(require '[malli.core :as m])
(defn pow
{:malli/schema [:=> [:cat :int] :int]
:malli/gen true} ; how do I find/use this generated function body?
[x]
(* x x))
(require 'malli.dev)
(malli.dev/start!)
(pow 1)
;1#2022-10-0913:43dvingoYou have to pass in malli.generator/generate to the start! call.
there is an example under this section: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#instrumentation ("With :gen we can omit the function body"..)
(defn pow {:malli/schema [:=> [:cat :int] :int] :malli/gen true} [x] (* x x))
(defn pow2 {:malli/schema [:=> [:cat :int] :int] :malli/gen false} [x] (* x x))
(md/start! {:gen malli.generator/generate})
(comment
(pow 45)
; => random int
(pow2 45)
; => 2025
)#2022-10-0916:52skynet@U051V5LLP perfect! just what I needed. didn't put it together to use the same argument to start!, thanks#2022-10-0917:47skynetoh and one missing piece: I now need to add :malli/gen false to the defn metadata in order to disable gen for specific functions when doing this, but this might work out#2022-10-0920:30dvingooh yea, I didn't realize that you have to opt out - another idea is to use the :filters option to target only those that you want to gen#2022-10-1110:05robert-stuttafordhas anyone had a go at writing malli specs with https://github.com/lambdaisland/regal ? what i like about this is that it makes the regex a little more self-documenting
edit: nevermind, there's a malli section right in the readme!#2022-10-1615:30ikitommiI haven’t updated the Regal-side in some time. please update if the integration is out-of-date#2022-10-1615:31robert-stuttafordwill do!#2022-10-1118:54Nikolas PafitisHi I have a schema that looks like this
[:map
...
[:timeline-type [:enum :reactive :proactive-actual :proactive-alternative]]]
And I'm trying to use this schema with malli coercion to coerce the query params, now from the frontend i send this timeline-type as a string (obviously as it's in the query string) but it's not coerced to a keyword. Is there some option for :enum (or any other type for that matter) that I can specify a specific coercer for reitit. For example something like:
[:enum {:malli.coerce/coerce-as :keyword} :reactive :proactive-actual :proactive-alternative]
or something along those lines? Or what else could I do here.#2022-10-1615:29ikitommiHi. There is an issue for resolving the real type from schemas, see https://github.com/metosin/malli/issues/264. Before that, you can use :and -> [:and :keyword [:enum :reactive :proactive-actual :proactive-alternative]]#2022-10-1119:39HankstenbergHi guys, quick question: If I run m/walk in clojurescript like this:
(m/walk
[:map [:id string?]]
(m/schema-walker identity)
I'm getting back a "#object[malli.core.t_malli$core35871]" instead of the vector I put in. What can I do with this object?#2022-10-1120:39dvingotry passing it to (malli.core/form to get the vector out if that's what you want#2022-10-1204:42HankstenbergOh, okay now I get it thanks!#2022-10-1204:36timothypratleyHello 👋
To include a literal, is it best to use :fn ?
In these examples I want to limit the matches to a specific value.
This seems to work:
((ma/parser [:fn #{1}]) 1)
;=> 1
((ma/parser [:* [:cat [:fn #{1}] [:fn #{2}]]])
[1 2 1 2 1 2])
;=> [[1 2] [1 2] [1 2]]
Does this seem like the correct approach?#2022-10-1212:49Chris O’DonnellI would probably use [:= 1] personally, but your approach seems fine, too.#2022-10-1420:57timothypratleyOh great thank you, I didn't see := that's what I was hoping for#2022-10-1605:18Ben Slessfn spec is best avoided IMO, it makes it hard to reason about specs#2022-10-1214:31Stig BrautasetCan I instrument protocol methods with Malli? We use protocols (implemented by records) quite a lot. With Spec we add an indirection via a regular function to have something to hang Clojure Spec specs off of. Is that what I’d do with Malli too?#2022-10-1317:15dvingoAmbrose was looking into this https://blog.ambrosebs.com/2022/09/08/schema-defprotocol.html
https://clojurians.slack.com/archives/C06MAR553/p1661105488092049#2022-10-1712:31Stig BrautasetThank you!#2022-10-1218:14PrashantHi, I was curious if anyone has any pointers/implementation on validating EntityMap using Malli . These are returned by (datomic.api/entity db entity-id) .
These EntityMap objects don't implement IPersistentMap so schema like [:map ...] fails with type mismatch.
I would greatly appreciate the inputs.#2022-10-1302:05dvingoYou can create your own schema type using -simple-schema (here are some examples: https://github.com/metosin/malli/blob/1a9b3767f1d64d504663ca151363244db2635708/src/malli/core.cljc#L660)
I'm not sure on the details but I think you can make your predicate function convert the entity to a hashmap and then leverage the existing :map schema type by having your custom schema type pass its arguments to the :map type#2022-10-1313:21PrashantThanks @U051V5LLP I will give it a shot.#2022-10-1314:06PrashantI have reached till below and was wondering how to pass the arguments to the :map type.
(defn entity?
[entity]
(instance? datomic.query.EntityMap entity))
(defn -entitiy-map-schema
[]
(-simple-schema
{:type :entity-map
:pred entity?
:type-properties {:decode/map #(if (entity? %)
(into {} %)
%)}}))
On a sidenote, since a instances of datomic.query.EntityMap may have nested datomic.query.EntityMap, I think using a ::ref would also be needed?#2022-10-1314:56dvingohmm, I think it might not be as simple as using -simple-schema
you may have to copy the -map-schema implementation and delegate to that, here is a heavily hacked one I got working but gives the general idea:#2022-10-1314:56dvingo(defn -entity-map-schema
([]
(-entity-map-schema {:naked-keys true}))
([opts] ;; :naked-keys, :lazy
^{:type ::into-schema}
(reify
m/IntoSchema
(-type [_] :entity-map)
(-type-properties [_])
(-properties-schema [_ _])
(-children-schema [_ _])
(-into-schema [parent {:keys [closed] :as properties} children options]
(let [map-schema (m/schema (into [:map] children))
entry-parser (m/-create-entry-parser children opts options)
cache (m/-create-cache options)]
^{:type ::schema}
(reify
m/AST
(m/-to-ast [this _] (m/-to-ast map-schema _))
m/Schema
(-validator [this]
(let [keyset (m/-entry-keyset (m/-entry-parser this))
_ (println "2 " (m/children map-schema))
validators (map
(fn [[key {:keys [optional]} value]]
(let [valid? (m/-validator value)
default (boolean optional)]
(fn [m] (if-let [map-entry (find m key)]
(valid? (val map-entry))
default))))
(m/children map-schema))
validate (apply every-pred validators)]
;; THIS LINE IS THE SIGNIFICANT CHANGE:
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(fn [m] (and
(instance? datomic.query.EntityMap m)
(validate (into {} m))))))
(-explainer [this path] (m/-explainer map-schema path))
(-parser [this] (m/-parser map-schema))
(-unparser [this] (m/-unparser map-schema))
(-transformer [this transformer method options] (m/-transformer map-schema transformer method options))
(-walk [this walker path options] (m/-walk-entries map-schema walker path options))
(-properties [_] properties)
(-options [_] options)
(-children [_] (m/-entry-children entry-parser))
(-parent [_] parent)
(-form [_] (m/-form map-schema))
m/EntrySchema
(-entries [_] (m/-entry-entries entry-parser))
(-entry-parser [_] entry-parser)
m/Cached
(-cache [_] cache)
m/LensSchema
(-keep [_] true)
(-get [this key default] (m/-get-entries this key default))
(-set [this key value] (m/-set-entries this key value))))))))#2022-10-1314:57dvingothe main thing is when malli creates the schema (using -into-schema you take the provided children and create an underlying :map schema
map-schema (m/schema (into [:map] children))
#2022-10-1314:57dvingothen fill in all the protocol methods for IntoSchema using that#2022-10-1314:58PrashantThanks a ton!!! this is super helpful.#2022-10-1314:58dvingothere may be a much simpler way to do this, I'm not an expert on this, just figuring things out by reading the source code of malli.core#2022-10-1315:00dvingoand you can try it with:
(m/validate
(m/schema [:entity-map [:x :int]]
{:registry (assoc (m/default-schemas) :entity-map (-entity-map-schema))})
(d/entity db [:some/id-prop 5]))#2022-10-1315:00PrashantThis gives a very good head start. I was also thinking on the same lines of copying map schema and hacking through it.#2022-10-1315:01PrashantThanks again @U051V5LLP!!!#2022-10-1315:02dvingosure thing! I'm still learning myself, so would be curious what you come up with 🙂#2022-10-1615:23ikitommiWould it help if the malli.core/-map-schema took an extra option for the predicate? e.g. on could just do:
(def EntityMap (m/-map-schema {:pred entity?}))
(m/validate EntityMap ...)#2022-10-1615:25ikitommiMalli is intended to be extendable, so all IntoSchemas are created using functions and it’s a easy & non-breaking change to add new options 🙂#2022-10-1615:26ikitommi(also, effects bundle-size on cljs, non-used schemas can be DCEd)#2022-10-1615:26ikitommianyway, PR welcome on adding the :pred for the m/-map-schema#2022-10-1714:10PrashantThanks for the suggestion Tommi.
I will definitely look into implementing your suggestion and when satisfied, raise a PR 🙂#2022-10-1715:45Prashant@U055NJ5CC Raised this https://github.com/metosin/malli/pull/767 per your suggestion 🙂#2022-10-1406:33robert-stuttafordwhere can i learn about how to write generators for complex :multi specs? currently butting my head against this:
(mg/sample ;; OK
[:map [:db/id {:optional true}]])
(mg/sample ;; OK
[:multi {:dispatch :type}
[:some.type/value
[:map
[:some/key :string]]]])
(mg/sample ;; Couldn't satisfy such-that predicate after 100 tries.
[:and
[:map [:db/id {:optional true}]]
[:multi {:dispatch :type}
[:some.type/value
[:map
[:some/key :string]]]]])#2022-10-1410:28pithylessTwo things come to mind:
1. I think your :multi map needs to include the dispatch key (otherwise it won't be generated correctly)
2. Is it feasible for you to merge the multi map? This would work:
(mg/sample
[:multi {:dispatch :type}
[:some.type/value
(mu/merge
[:map
[:type [:= :some.type/value]]
[:some/key :string]]
[:map
[:db/id :string]])]])#2022-10-1410:29pithylessNevertheless, subscribing to this thread in hope that there is a better way to solve this problem (have encountered it as well).#2022-10-1410:32robert-stuttafordthanks that gives me some things to try 😅#2022-10-1415:39Stig BrautasetThis is a topic I’m interested in too. I tried to spec out the complete state of a board game (a toy project), and it was very easy to get into a situation where the generation failed.#2022-10-1507:21robert-stuttafordok so it seems i can't use mu/union because of cyclic dependencies between my specs; so it looks like generating data is off the table for me here.
just in case there's something i'm missing, if you've any advice @U055NJ5CC, i'd greatly appreciate it 🙂#2022-10-1605:15Ben SlessYou can break cyclic dependencies with a ref spec#2022-10-1607:28robert-stuttafordi am using ref specs - the issue is that to use mu/union, the keyword specs you give it have to be registered already, which forces a specific code loading order, but i can't ensure that order because the spec is for an entity that links to other entities of the same spec, linked-list type 'next-item' / 'parent-item' stuff.
the specs work as validation (with :ref specs); it just doesn't work to generate#2022-10-1607:32Ben SlessHuh, weird. Want to send the complete schema (or minimal repro)? I think I managed that once#2022-10-1607:49robert-stuttafordapex spec is :multi. all of the internal impls all a shared base :map spec, and some are that base plus some extra :map spec. the base :map spec has :refs to the apex :multi spec.
then, after all that, i am also wrapping this in a generic Datomic id handling system, where whenever i specify a :db.type/ref, there's a wrapper spec that allows either a long, a tempid, a map with a long or tempid, or an :and of :map :db/id optional and whatever the actual spec was; in this case, the apex multi spec.
all of this validates, but it doesn't generate 😬
i suspect what i want is conceptually not possible, but i don't know enough about malli or the domain it operates in to know for sure.#2022-10-1607:50robert-stuttafordi can provide pseudocode if that was too eye-watering to reason through 🙂#2022-10-1608:08Ben SlessHave you tried using a schema schema to wrap it all together?#2022-10-1608:08Ben Slesse.g. [:schema {:registry everything-you-just-said} ::apex]?#2022-10-1608:13robert-stuttafordah, i am incrementally building this up with calls to swap! on an atom that has been given to (mr/set-default-registry! (mr/mutable-registry *registry)) . i'll work up a minimal repro version of what i said without that and with [:schema {:registry ...} ::apex] and see how it goes!#2022-10-1615:17ikitommiHi, some comments on this:
• :and generator generates on the first and just validates the generated result with the validators of the rest of the childs -> not likely to match
• I was sure that there was an example on :multi + gen in README, but it seems there is not! but for now, there is no logic to merge the dispatch types into the schemas, so you have to do it yourself. Here’s an example from README:
[:multi {:dispatch :type
:decode/string #(update % :type keyword)}
[:sized [:map [:type [:= :sized]] [:size int?]]]
[:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
• should find time to work on https://github.com/metosin/malli/issues/264, would help here and I think this is really important anyway
• if all childs are maps and the :dispatch value is a Keyword or a String, there could be a helper to push the key + value into the maps. this helper could be used in gen, json-schema etc. namespaces to make it work correctly
• I’m thinking :and is bad (like Plumatic Team found out) and there should be something like :constrained instead.
• sorry @robert-stuttaford, I guess this didn’t help you, but that’s something for the background. Looking forward to your repro + Ben’s solution 🙂 #2022-10-1615:30robert-stuttafordthanks @U055NJ5CC, this is useful info!#2022-10-1616:53Ben SlessBased on some experience playing with kanren, should and implement a bind of one generator to the next?#2022-10-1617:12Ben SlessNot sure if it's viable, but a schema can describe how it can return a generator from a seed value to build up with bind. That way, for example, a map schema would use merge#2022-10-1615:17ikitommiHi, some comments on this:
• :and generator generates on the first and just validates the generated result with the validators of the rest of the childs -> not likely to match
• I was sure that there was an example on :multi + gen in README, but it seems there is not! but for now, there is no logic to merge the dispatch types into the schemas, so you have to do it yourself. Here’s an example from README:
[:multi {:dispatch :type
:decode/string #(update % :type keyword)}
[:sized [:map [:type [:= :sized]] [:size int?]]]
[:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
• should find time to work on https://github.com/metosin/malli/issues/264, would help here and I think this is really important anyway
• if all childs are maps and the :dispatch value is a Keyword or a String, there could be a helper to push the key + value into the maps. this helper could be used in gen, json-schema etc. namespaces to make it work correctly
• I’m thinking :and is bad (like Plumatic Team found out) and there should be something like :constrained instead.
• sorry @robert-stuttaford, I guess this didn’t help you, but that’s something for the background. Looking forward to your repro + Ben’s solution 🙂 #2022-10-1618:37ikitommi… and, it’s out 🥳. Thank’s everyone!!#2022-10-1721:38skynetI'm seeing a bug in malli 0.9.0 and 0.9.1, and I'm trying to get a minimal reproduction of this issue but I can't. the error I'm getting is:
Exception: java.lang.ClassCastExcetpion: class clojure.lang.Symbol cannot be cast to class java.lang.String (clojure.lang.Symbol is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
at clojure.core$symbol.invokeStatic (core.clj:591)
...
malli.instrument$_strument_BANG_$fn__24440$fn__24441.invoke (instrument.clj:23)
malli.core$_instrument$fn__21304.doInvoke (core.cljc:2480)
...
<my code>
this occurs in a test after doing !. there are some functions in the project with :malli/gen true in function metadata, and some with that set to false. malli is started with:
(malli.dev/start!
{::m/function-checker mg/function-checker
:report (pretty/reporter)
:gen mg/generate})
this all works on 0.8.9. unfortunately I can't share the code, but I'm not sure this is enough for someone to fix it#2022-10-1722:32skynetoh actually I figured it out, I was redefining a fn that was instrumented, and had it returning invalid data, like this https://github.com/skynet-gh/2022-10-17-malli-instrument-bug
then malli calls (symbol n s) in that case https://github.com/metosin/malli/blob/master/src/malli/instrument.clj#L23
and n and s are symbols in that case, which doesn't work for multi-arity https://clojuredocs.org/clojure.core/symbol
so changing it to (symbol (name n) (name s)) works#2022-10-1722:43skynetok I over-complicated it. I think it will break after ! with :report whenever you have any invalid input/output on an instrumented function#2022-10-1722:50dvingoOh shoot, I just touched that code recently, I can take a look#2022-10-1723:29dvingohttps://github.com/metosin/malli/pull/768
TIL - that code works in cljs !#2022-10-1813:24skynetthat's tricky, I didn't know that either#2022-10-1805:58ikitommi[metosin/malli "0.9.2"], thanks @danvingo for the quick fix!#2022-10-1805:59ikitommiwould be good to have tests for the dev-tooling too for the future.#2022-10-1812:51dvingoagreed! I will take a look into adding some for start!#2022-10-1822:15dvingo@ikitommi I'm writing some tests for ! and friends and noticed that the arguments are a little confusing.
start! takes :ns which indicates the set of namespaces to be collected, but then instrument! doesn't use that set - instead you have to separately pass :filters with [(mi/-filter-ns...)
do you think it makes sense to use the set of namespaces passed in to start! if they are present and pass those to filters so instrument! will use the same set? Otherwise you have to pass both :ns and :filters to only start collection and instrumentation on the desired set of namespaces#2022-10-1905:01ikitommi:thinking_face: just looked at the code, dev/start! doesn't seem to use any :ns option either. Where did you pick that up?#2022-10-1911:29dvingooh wow, that's my mistake!
I see what happened - it's collect! that uses it,
https://github.com/metosin/malli/blob/546eb663484e66ef1271b99189b17dbeca215ec8/src/malli/dev.clj#L23
which does support changing what is collected:
https://github.com/metosin/malli/blob/546eb663484e66ef1271b99189b17dbeca215ec8/src/malli/instrument.clj#L46
I must have saw that and inferred that you wouldn't always want to collect everything when using dev/start!
do you have any thoughts on if the :ns option should be supported for start!? I can change it back for the cljs version if you want#2022-10-2006:33ikitommiI think it should in start! too, but silly to be only on collect!#2022-10-1903:40JoelUsing the subschemas example, I wanted to pull out some custom data from the schema structure with the associated path. From something like this… [:street {:optional true :custom-field :value} string?] I want to make grab the path and that :value I put in that the properties.
However, I see that using mu/subschemas does not return the properties. I guess I need to write my own walker? I find that puzzling since I think malli would need that :optional true info.#2022-10-1904:57ikitommicould you paste a minimal repro here?#2022-10-1913:44Joel(def Schema
(m/schema
[:map
[:purchases {:optional true } [:set string?]]]))
(mu/subschemas Schema)
When :purchases is walked, it doesn’t list the {:optional…}#2022-10-1914:37ikitommiIt’s because the properties are on the map-entry, not the child itself.#2022-10-1914:37ikitommithere is mu/find that works like clojure.core/find, e.g. returns the [key properties value] entry.#2022-10-1914:37ikitommiso, you can:
(def Schema
(m/schema
[:map
[:purchases {:optional true} [:set string?]]]))
(defn get-in-props [schema path]
(if-let [?entry (mu/find (mu/get-in schema (butlast path)) (last path))]
(when (vector? ?entry) (second ?entry))))
(for [sub (mu/subschemas Schema)]
(assoc sub :entry-props (get-in-props Schema (:path sub))))
;({:path [], :in [], :schema [:map [:purchases {:optional true} [:set string?]]], :entry-props nil}
; {:path [:purchases], :in [:purchases], :schema [:set string?], :entry-props {:optional true}}
; {:path [:purchases :malli.core/in], :in [:purchases :malli.core/in], :schema string?, :entry-props nil})#2022-10-1914:37ikitommihope that helps.#2022-10-1914:38Joellooks like it will, ill give it a go.#2022-10-1914:44Joeli don’t understand what :malli.core/in is “about”, but this gets me what i need - thanks!#2022-10-1914:54ikitommiit's basically a pointer into to a homogeneous seq, "values in any position".#2022-10-1906:28timothypratleyHi 👋
I'm trying to understand 2 things about catn:
'[:catn
[m [:schema [:ref "m"]]]
[s [:schema [:ref "s"]]]
[_ [:orn [v [:schema [:ref "v"]]] [k [:schema [:ref "k"]]]]]]
1. It seems to produce the same output whether I use :orn or :altn -- is there a reason to prefer one over the other?
2. The result it produces is {m {:a 1}, s #{:a}, _ [k :k]} is there any way I can produce {m {:a 1}, s #{:a}, k :k} instead? Where the catn takes the name from the choice operator (`:orn`)?
Below is the full schema but you can ignore most of it, it's just the choice between vector and keyword at the end that I'm focusing on:
'[:schema
{:registry {"start" [:and vector?
[:catn
[m [:schema [:ref "m"]]]
[s [:schema [:ref "s"]]]
[_ [:orn [v [:schema [:ref "v"]]] [k [:schema [:ref "k"]]]]]]],
"m" map?,
"s" set?,
"v" [:and vector?
[:catn [a [:schema any?]] [b [:schema any?]] [c [:schema any?]]]],
"k" keyword?}}
"start"]
And an example input:
'{m {:a 1}
s #{:a}
k :k}
#2022-10-1920:08timothypratleyOne thing I discovered is that I can move the name binding "inside" cat using single argument orn instead of using catn:
[:cat
[:orn [m [:schema [:ref "m"]]]]
[:orn [s [:schema [:ref "s"]]]]
[:altn [v [:schema [:ref "v"]]] [k [:schema [:ref "k"]]]]]
=> [[m {:a 1}] [s #{:a}] [k :k]]
Which is interesting.#2022-10-1923:18timothypratleyWhy does using a schema ref catn behave differently from an embedded catn:
(ma/parse
'[:schema {:registry {"start" [:and vector? [:catn
[a [:schema any?]]
[b [:schema [:ref "b"]]]]],
"b" [:catn [c [:schema any?]] [d [:schema any?]]]}}
"start"],
'[1 (2 3)])
(ma/parse
'[:schema {:registry {"start" [:and vector? [:catn
[a [:schema any?]]
[b [:catn [c [:schema any?]] [d [:schema any?]]]]]]}}
"start"],
'[1 2 3])
^^ The first (using a schema ref) matches a list inside a vector [1 (2 3)], whereas the second (embedded) matches a flat vector [1 2 3] -- I'm wondering if there is a way to use a ref to match a flat vector instead of a list.#2022-10-2006:22ikitommitry to answer both here:
• if you you a regex schema inside a regex schema, it get’s inlined, so you can create a “flat” list using those
• non-regex schemas just take one slot in the regex schema, so:
◦ [:orn [:s :string] [:bs [:+ :boolean]] is “a string or 1+ booleans” taking 1 slot
◦ [:altn [:s :string] [:bs [:+ :boolean]] is the same but if used inside a regex schema, the :+ get’s inlined
◦ you can’t inline a :ref in the regex schema
◦ wrapping any schema into :schema makes it “normal schema” and thus, it takes just one slot in the regex (`:ref` and regex schemas)#2022-10-2020:30timothypratleyI see, thank you 🙂#2022-10-2020:05skynetwhen I'm using malli in a terminal like setting, I kind of want the pretty explain output reversed from how it is seen on a webpage: https://github.com/valyagolev/malli/blob/c7c22a8fefd3a8491a35e1e320ab620791c4500c/src/malli/dev/pretty.cljc#L37-L40
since the bottom is closest to what I'm looking at, and the top stuff I might have to scroll up to see. luckily I'm able to call this whole defmethod myself, but maybe it's common enough that other people would find it useful to have a function in that namespace to set explain to "console" mode?#2022-10-2106:55ikitommiconsole-mode sound good. Also, color themes (none/light/dark/custom) + html-emitting mode (for react error boundaries etc). How different would the :console mode be from the normal? All mm-methods should be different? just the one? some amount of config can be done via adding configuration options to the printer (it’s backed by a map), but if it’s totally different (the html-mode could be), then another mm & ns might be better.#2022-10-2113:12skynetI'm currently just rearranging the blocks (and removing the "more info" block):
(defmethod v/-format ::m/explain
...
(-block "Schema:"
(-block "Value:"
(-block "Errors:"
only other thing I might change is the amount of whitespace. there's like 3 blank lines after the error, I might reduce to 1 line. mayyybe remove blank lines after the Schema:, Value:, Errors: section titles and content, but that could make it less readable#2022-10-2106:55ikitommiconsole-mode sound good. Also, color themes (none/light/dark/custom) + html-emitting mode (for react error boundaries etc). How different would the :console mode be from the normal? All mm-methods should be different? just the one? some amount of config can be done via adding configuration options to the printer (it’s backed by a map), but if it’s totally different (the html-mode could be), then another mm & ns might be better.#2022-10-2112:16Thomas MoermanI'm currently implementing a custom schema (for reference types).
My question: Is it idiomatic to call (m/validate ?schema ?subject {:registry ?my-local-reg}) within the body of a content-dependent schema like in:
(let [;; self-checking keyword spec is necessary to make `[:my/ref :asset/id] work
local-registry
(merge
;; self-validating schemas
{:asset/id (m/schema [:fn #(= :asset/id %)])
:grommet/id (m/schema [:fn #(= :grommet/id %)])}
;; for ID values
{:or (m/-or-schema)
:int (m/-int-schema)
:string (m/-string-schema)
:uuid (m/-uuid-schema)
:id [:or :int :string :uuid]})
;; content-dependent schema
Ref (m/-simple-schema (fn [props children]
(log/spy :info children)
{:type :my/ref
:pred (fn [x]
(let [attr-schema (first children)
[id-attr id-val] (first x)]
(and (map? x)
(= 1 (count x))
;; ============== ;;
;; Q: is this ok? ;;
;; ============== ;;
(m/validate :id id-val {:registry local-registry})
(m/validate attr-schema id-attr {:registry local-registry}))))
;; Don't forget the children count constraints!
:min 1
:max 1}))
custom-reg {:my/ref Ref}
registry (mr/composite-registry
m/default-registry
local-registry
custom-reg)]
[(m/validate [:my/ref :asset/id] {:asset/id 123} {:registry registry})
(m/validate [:my/ref :asset/id] {:grommet/id 123} {:registry registry})
(m/validate [:my/ref [:or :asset/id :grommet/id]] {:asset/id 123} {:registry registry})
(m/validate [:my/ref [:or :asset/id :grommet/id]] {:other/id 123} {:registry registry})
])
Is there perhaps another approach preferable?
Thanks 🙏#2022-10-2112:42Thomas MoermanI guess a better version would look somethat like -maybe-schema instead of using simple-schema#2022-10-2207:46ikitommiHave stumbled few times on writing custom reference schemas, not trivial. No best practices, but we have been adding new configuration options to schemas, (e.g. just added :pred to m/-map-schema), so if there is an existing schema type which is almost what you need and would be exactly what you need with adding an configuration option - I’m open to hearing & maybe adding that.#2022-10-2207:47ikitommibut, prefer using a cached m/validator instead of m/validate if possible, it’s much faster.#2022-10-2214:36Thomas MoermanYeah you're right, it's not trivial. I'm trying to get my head around the essence of what makes it so difficult, there is some kind of circularity in the problem, it seems.#2022-10-2207:42ikitommiShould m/-instument instrument also nested function definitions by default? https://github.com/metosin/malli/issues/770#2022-10-2400:53aaron51Does Malli have a way to wrap input in a vector, if it’s not already a vector? I know this is a one-liner in Clojure but it’s nice to keep all the coercion together.
(if (vec? x) x [x])#2022-10-2404:50ikitommidon’t think so. what is your use case for this? input coercion?#2022-10-2404:51ikitommithere is mt/collection-transformer https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L416-L423, but doing just just any-seq-like->target-type, not wrapping non-seqs into collections.#2022-10-2410:40aaron51Use case is a reitit handler that accepts one or more than one multipart files. When the request has one file, the param is a map. When multiple files, the param is a vector. Would be a bit easier to handle if it was always a vector.#2022-10-2411:20ikitommiI see, you can create a custom transformer for this. like the mt/collection-transformer but with custom functions. You can map any transformer into reitit coercion. not fun, but doable.#2022-10-2614:35bortexzIs this a bug, or am I doing something wrong?
(defonce registry* (atom (m/default-schemas)))
(mr/set-default-registry! (mr/mutable-registry registry*))
(defn sdef
"Defines a new schema in mutable [[registry*]]."
([type schema] (swap! registry* assoc type schema) schema)
([type props schema] (swap! registry* assoc type [:schema props schema]) schema))
(sdef ::thing-nested2 int?)
(sdef ::thing-nested (mu/optional-keys [:map ::thing-nested2]))
(sdef ::thing (mu/optional-keys [:map ::thing-nested]))
(mu/update ::thing ::thing-nested (fn [s] (mu/required-keys s [::thing-nested2])))
Blows up with:
; Evaluating file: malli.clj
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/invalid-schema {:schema nil}
; Evaluation of file malli.clj failed: class clojure.lang.Compiler$CompilerException#2022-10-2616:34bortexzIf reading the schema from the registry directly, it seems to work:
(mu/update (get @registry* ::thing) ::thing-nested (fn [s] (mu/required-keys s [::thing-nested2])))
#2022-10-2617:10ikitommihmm. that’s not good.#2022-10-2617:29ikitommiThe malli.util helpers do not deref the schemas => calling mu/get on a reference tries to get from the reference, not from the value behind it.#2022-10-2617:30ikitommicalling m/deref-all on the subject should work, but just for one level. This is unfortunate.#2022-10-2617:31ikitommichecked if that’s easy to change in malli, doesn’t seem to: https://github.com/metosin/malli/pull/772/files - all tests fail 😞#2022-10-2617:33ikitommihere’s the list of all paths that are part of the schema:
(mu/subschemas ::thing)
;[{:path [],
; :in [],
; :schema :malli.core-test/thing}
; {:path [0]
; :in []
; :schema [:map [:malli.core-test/thing-nested {:optional true} :malli.core-test/thing-nested]]}
; {:path [0 :malli.core-test/thing-nested]
; :in [:malli.core-test/thing-nested]
; :schema :malli.core-test/thing-nested}
; {:path [0 :malli.core-test/thing-nested 0],
; :in [:malli.core-test/thing-nested],
; :schema [:map [:malli.core-test/thing-nested2 {:optional true} :malli.core-test/thing-nested2]]}
; {:path [0 :malli.core-test/thing-nested 0 :malli.core-test/thing-nested2],
; :in [:malli.core-test/thing-nested :malli.core-test/thing-nested2],
; :schema :malli.core-test/thing-nested2}
; {:path [0 :malli.core-test/thing-nested 0 :malli.core-test/thing-nested2 0],
; :in [:malli.core-test/thing-nested :malli.core-test/thing-nested2],
; :schema int?}]#2022-10-2617:34ikitommie.g. each reference schema contributes to the path with 0.#2022-10-2617:35ikitommiso, this would work too:
(mu/update-in ::thing [0 ::thing-nested] (fn [s] (mu/required-keys s [::thing-nested2])))
#2022-10-2617:36ikitommiplease write an issue, will think about this.#2022-10-2707:43bortexzDone (https://github.com/metosin/malli/issues/773) thanks for looking into this @U055NJ5CC 🙏#2022-10-2717:27souenzzodoes malli has something like spec's conform operation?
my use case is:
my schema has an or
:a string?
:b int?
:ab [:or :a :b]
And I a function that given the :ab schema + a value, it tells me if the value is :a, :b or :invalid#2022-10-2717:35ikitommitry m/parse & m/unparse. If you want named nodes, you can: [:orn [:s :string] [:b :boolean]]#2022-10-2717:37ikitommisame with regexs, adding a n to the name, you can present the nodes in entry-style and get named results, e.g. :cat -> :catn, :alt -> :altn#2022-10-2717:47souenzzonice! TY#2022-10-2721:48mauricio.szaboFolks, can I ask for a huge improvement on Malli? Show the errors on instrument, on .pretty/explain, etc, at the end of everything#2022-10-2721:48mauricio.szaboSometimes when I get an error, I can't find it anywhere - either it is greater than my terminal buffer or it gets lost in the middle of a "your state is this" and "your schema is this"#2022-10-2721:57mauricio.szaboIt's not just inconvenient - on React-Native, for example, sometimes the console is not redirected to the REPL (because metro and other issues) so I keep having to disable and re-enable instrumentation 😢#2022-10-2814:26dvingothe reporter option can be passed in to instrument and to start, so pretty quick work to copy the pretty namespace to your project and mess around how you like#2022-10-2815:27mauricio.szaboI... actually have no idea what you mean 😄#2022-10-2815:27mauricio.szaboDo you have an example?#2022-10-2818:23dvingo(malli.dev/start! {:report (fn [type data] (.error js/console "Instrument error: " (str type)))})#2022-10-2818:25mauricio.szaboThanks, I'll try 🙂#2022-10-2818:27dvingosure thing 🙂 https://github.com/metosin/malli/blob/537c5fb8f96c35fc9e0d41629faa88a1aa8b56c3/src/malli/dev/pretty.cljc#L82 I meant you can copy this namespace and change it up as you like#2022-10-2723:02mauricio.szaboOk, for .pretty/explain there is a work-around - I can capture the :errors property and pretty-print them#2022-10-2809:57ikitommiAnyone interested in doing / sharing a Schema for def syntax? like the https://github.com/metosin/malli/blob/master/src/malli/destructure.cljc#L11-L55 here.#2022-10-2810:00ikitommiactually… this: https://github.com/metosin/malli/blob/master/src/malli/experimental.cljc#L8-L33#2022-10-2810:01ikitomminevermind, already working on it 🙂#2022-10-2814:24dvingoif you're touching that code, i think there is an edge case for cljs (maybe clj too) if you have a custom registry - the schemas used for parsing the experimental defn form may not be available -
https://github.com/metosin/malli/pull/702/files#diff-8fa8701bc52f9f699e5512b8ea028a44c05033b134751af5f789cd57edcf05ad
had to add that line to get it working.#2022-10-2815:15sashtonAre closed-schema and multi incompatible? I am calling closed-schema on a schema which includes a multi nested in it. It fails to validate correctly, but if I remove the closed-schema , it works as expected.
Example:
(def operator-schema
(malli/schema
[:multi {:dispatch :op}
[:addition [:map
[:summand-1 :int]
[:summand-2 :int]]]
[:multiplication [:map
[:factor-1 :int]
[:factor-2 :int]]]]))
(def equation-schema
(mallu/closed-schema
[:map
[:left-hand-side operator-schema]
[:right-hand-side :int]]))
(malle/humanize (malli/explain equation-schema {:left-hand-side {:op :addition
:summand-1 1
:summand-2 2}
:right-hand-side 3}))
=> {:left-hand-side {:op ["disallowed key"]}}
#2022-10-2815:56sashtonNever mind, user error: I forgot to include the [:op :keyword] declaration in each of the submaps in operator-schema#2022-10-3109:49dumrat(me/humanize
(m/explain
[:and
[:fn (fn [{:keys [tag]}] (= :Portfolio tag))]
...]
{:tag :some-other-tag ...}))
=>
["unknown error"]
I understand that the "unknown error" is the result because malli doesn't know what to say when my custom function returns false. Can I customize this error message here?#2022-10-3109:50dumratI know I can use :enum here to do the same validation. Just want to know how to handle custom fns.#2022-10-3109:52ikitommi(me/humanize
(m/explain
[:and
[:fn {:error/message "horror"}
(fn [{:keys [tag]}] (= :Portfolio tag))]]
{:tag :some-other-tag}))
; => ["horror"]#2022-10-3109:53dumratgreat thanks#2022-10-3109:53ikitommiyou’re welcome#2022-10-3115:12dumratData:
[{:key1 1 :val 4} {:key2 2 :val 5} {:key3 3 :val 6}]
I'd like to write a schema which requires all three keys key1, key2, key3 to be present in the sub maps of the input vector. How to do this?#2022-11-0102:51dumratIs there a schema for malli schemas? 😄#2022-11-0105:29ikitommithere is support for building that 🙂 e.g. in IntoSchema protocol, there are:
(-properties-schema [this options] "maybe returns :map schema describing schema properties")
(-children-schema [this options] "maybe returns sequence schema describing schema children")
… just have had no time to add the implementations to those. Validation of a form can be done with m/schema, throws if not a valid thing.#2022-11-0116:11dumratAny reason why malli validation would not honor {:optional true} ?
i.e.
[:map
[:details
[:map
[:email {:optional true} :string]]]]
Proceeds to ignore the optional status when validating and fails validation.#2022-11-0116:31timothypratley(ma/validate [:map
[:details
[:map
[:email {:optional true} :string]]]]
{:details {}})
=> true#2022-11-0116:32timothypratleyseems fine?#2022-11-0208:45dumratMy bad. I needed to use ‘maybe’ here.#2022-11-0208:47dumratIs there a way to use a schema which has few maybe schemas but consider maybe not be there for validation? Or do I have to strip the maybes from schema and do validation manually? On mobile, I can give code sample later if needed.#2022-11-0209:06ikitommiplease share an example#2022-11-0303:42dumratok case is like this.
1. I've got a few endpoints (around 20) returning large sets of xml. These are quite old services that we don't have control over.
2. I parse xml, then do some transformations on them
3. I have malli schemas for the transformed data. I coerce the data using schemas. Then I validate using same schemas.
4. Data from the services are often incomplete (nil values where they shouldn't be mostly). So I add [:maybe ...] for some fields to sort of make the validations work.
5. Now, I'd like to still know the cases where validations fail without the [:maybe ...] wrappings so I can have a report of missing stuff. Is it possible to do this or do I have to transform schema myself and remove the [:maybe ...] wrappings?
sample:
(def counterparties-schema
[:vector
[:map
[:address [:maybe :string]]
[:baseNumber :int]
[:contactName [:maybe :string]]
[:contactPhone [:maybe :string]]
[:cptyType [:maybe :string]]
[:enabledForSettlement :boolean]
[:id :int]
[:legalVehicleId :int]
[:mnemonic :string]
[:name [:maybe :string]]]])
;; Validation with this succeeds
(m/validate counterparties-schema data)
=> true
;; But I'd still like to know fields where validation would have failed if [:maybe ...] wrappings weren't there.
;; i.e [:address :string] etc.
;; Manually something like this:
(defn strict [schema]
(clojure.walk/postwalk
(fn [v]
(if (and (vector? v) (= (first v) :maybe))
(second v) v))
schema))
;; So I can still know the missing values
(m/validate (strict counterparties-schema) data)
What I was asking first is that is this kind of thing supported by malli itself? Perhaps that's a dumb question.
Is this a common use case though? Or am I thinking about this wrong?#2022-11-0209:07ingesolCLJS instrumentation reloading discussion in #clojurescript https://clojurians.slack.com/archives/C03S1L9DN/p1667377689417939?thread_ts=1667373586.775569&cid=C03S1L9DN#2022-11-0217:39mauricio.szaboA question about decompleted map keys and values: if I register, for example, that :user/id is always an [:string {:min 1}], is there a way to query Malli for :user/id and know the result is [:string {:min 1}]? Or even if it's just a :string?#2022-11-0218:19ikitommi(m/schema :user/id) returns the ref, you can deref that with m/deref (or m/deref-all)#2022-11-0221:11mauricio.szaboThanks, I had to do one step further to be useful:
(m/ast (m/deref-all (m/schema :user/id)))
That helped 🙂#2022-11-0308:17jeroenvandijkDid someone already use rewrite-cljc to point out malli validation errors inline? I believe it is possible and I’m thinking of doing this#2022-11-0309:14jeroenvandijkAll little bit how expound underlines invalid values with ^^^ (see https://github.com/bhb/expound#expound-1), so a little different than .pretty/explain i think. More like the output of babashka with for instance:
echo '(/ 100 0)' > foo.clj; bb foo.clj
....
----- Context ------------------------------------------------------------------
1: (/ 100 0)
^--- Divide by zero
#2022-11-0309:40ikitommiI would also like to see those. Also, paiting the errors with pretty, like expound does it.#2022-11-0309:41ikitommidrafted mx/def on the way to ClojureDays, which could have a clj-kondo hook to highlight the errors in the editor: https://twitter.com/ikitommi/status/1585995979748409345. ping @borkdude#2022-11-0309:42borkduderewrite-cljc is now rewrite-clj maintained under org.clj-commons#2022-11-0309:42jeroenvandijkah yeah made a mistake rewrite-clj indeed#2022-11-0309:43jeroenvandijkHowever just found that I can probably do what I want with edamame as well#2022-11-0309:43jeroenvandijkNeeded this https://github.com/borkdude/edamame#postprocess for detailed location info#2022-11-0309:45ikitommidid a test with edamame a while back: https://gist.github.com/ikitommi/0e5c4e48d8aeb7dd176128856ecdacb5#2022-11-0309:48jeroenvandijkAh useful thank you!#2022-11-0309:49borkdude@ikitommi I think we could add "type checking" for def - this is what you need right?#2022-11-0309:54ikitommi@borkdude “type-checking” using clj-kondo type system, yes, that would be awesome!#2022-11-0309:55ikitommianother try would be use use malli for the validation and just emit errors. But that’s way more work I think (all the relevant schemas should be loaded, e.g. should need to run the whole program to figure our what is the schema)#2022-11-0309:55ikitommiso, should the mx/def emit clj-kondo types?#2022-11-0310:07borkdude@ikitommi when clj-kondo supports this yes, but currently I don't think it supports it yet#2022-11-0310:07borkdudelet's see if there is already an issue for it#2022-11-0310:10borkdudeah yes, a very old one :rolling_on_the_floor_laughing:
https://github.com/clj-kondo/clj-kondo/issues/609#2022-11-0310:10borkdudeno upvotes ;)#2022-11-0310:15ikitomminow, one! 🙂#2022-11-0310:16ikitommibut, would that work here? the mx/def says “here’s a var which should have a value of type XYZ”#2022-11-0310:16borkdudethere is no support for this yet, I think#2022-11-0310:16borkdudeonly for functions#2022-11-0310:16borkdudeor perhaps there is#2022-11-0310:17ikitommicould hooks work here?#2022-11-0310:17borkdudemaybe try {:tag :int} or so#2022-11-0310:17ikitommi:thinking_face:#2022-11-0310:17borkdudeI have to go now, be back in an hour#2022-11-0310:17ikitommi:tag takes a clj-kondo type?#2022-11-0310:17ikitommisame here#2022-11-0310:17borkdudeyes: {:namespaces {foo {bar {:tag :int}}}} - if you are lucky this works, not sure#2022-11-0314:29jeroenvandijkFYI @ikitommi, I have something that I’m happy with (@borkdude also thanks to edamame), see https://gist.github.com/jeroenvandijk/080370966fadb7e65601931c3de47ed5#file-malli_inline_output-clj-L129. Here is some example output:
1: {:b :z
^------- should be an integer
2:
3: :c {:c0 2}
^------- missing required key :x
4:
5: }
^------- missing required key :a#2022-11-0314:34lreadOh that's pretty neat @U0FT7SRLP. I might experiment with this strategy for reporting errors in cljdoc.edn files.#2022-11-0314:43borkdude@U0FT7SRLP Cool stuff, I think it's worth a blog post or maybe an article in the malli or edamame repo :)#2022-11-0314:48jeroenvandijkThanks @borkdude! Yeah I guess it can be useful for others, or in other contexts. Also it is combining some funny techniques together. Will think about an article. Do you have an example format you are thinking of?#2022-11-0314:49jeroenvandijkBtw, I have created a similar mechanism for giving feedback on invalid edn input, also using Edamame. Maybe can include this in the same article#2022-11-0314:50borkdudeI'm open to having an examples/malli/README.md thing in edamame for example, or similar, or doc/postprocess.md or so#2022-11-0314:50borkdudebut if you want to post it in your own blog, also fine, then we can link there#2022-11-0314:51borkdudeIf you don't have a blog, you can quickly create one with ...
https://github.com/borkdude/quickblog :)#2022-11-0314:51jeroenvandijkhaha good one, let me think about it for a bit. No blog at the moment. I’ll come back to it#2022-11-0314:52borkdudeno pressure#2022-11-0315:11ikitommiLooks great! Also would like see both the code and the post 👍#2022-11-0315:13jeroenvandijkThanks @ikitommi a raw version is already here https://gist.github.com/jeroenvandijk/080370966fadb7e65601931c3de47ed5#file-malli_inline_output-clj-L129 (for the Malli feedback)#2022-11-0411:31borkdude@ikitommi https://twitter.com/borkdude/status/1588493830961393664
(WIP)#2022-11-0411:33borkdude(Need to fix a few test...
Ran 366 tests containing 3188 assertions.
80 failures, 8 errors.
)#2022-11-0411:39ikitommioh, super! that was fast. I’ll emit the clj-kondo types when this ships. So, this would work with complex types like maps too and also for defining the values? e.g. “this var should hold a map of :age key with int value” and saying (def mything {:age "1"}) would be a failure?#2022-11-0411:39borkdudeI'll have to test it :)#2022-11-0411:54borkdudeGetting closer:
Ran 366 tests containing 3233 assertions.
1 failures, 0 errors.
#2022-11-0411:57borkdudeThis is obviously wrong facepalm#2022-11-0411:58borkdudeRepro:#2022-11-0412:02borkdudeRan 366 tests containing 3233 assertions.
0 failures, 0 errors.
🎉#2022-11-0412:04borkdudeStill getting false positives with test corpus... lunch time#2022-11-0412:06borkdudeIt's funny, I'm getting errors from dynamic vars which are initialized to nil
and then later called with swap! - I guess I'll have to ignore dynamic vars#2022-11-0412:18borkdudehttps://twitter.com/borkdude/status/1588505774833426433#2022-11-0413:11borkdudeAnother nice edge case...
https://github.com/clojure/clojurescript/blob/961807166c8cf4d45a225d63416f06464fb27eaf/src/main/cljs/cljs/core.cljs#L10803-L10816
It thinks gensym_counter is nil (since it was initialized to nil)#2022-11-0413:11borkdudeso swap! is giving a type error#2022-11-0413:51borkdude@ikitommi Now got this working:
$ clojure -M:clj-kondo/dev --lint - --config '{:linters {:type-mismatch {:namespaces {user {x {:tag {:op :keys :req {:x :any}} }}}}}}' <<< "(def x) (inc x)"
<stdin>:1:14: warning: Expected: number, received: map.#2022-11-0413:51borkdudeSo here we declare that x is a map that has at least an :x key#2022-11-0413:54borkdudebut this is probably the reverse of what you were proposing. the config in clj-kondo is to override types that have been inferred, possibly incorrectly#2022-11-0414:00borkdudePerhaps clj-kondo should both check the type and take the config as the source of truth...?#2022-11-0414:00borkdudeI'm not sure how one would use that in practice#2022-11-0309:53JHi guys! Did you know if there a lib to convert a plumatic schema to a malli schema?#2022-11-0312:38jeroenvandijkNot sure if it exists, but maybe the lite syntax can help you convert your plumatic schema https://github.com/metosin/malli#lite#2022-11-0314:29jeroenvandijkFYI @ikitommi, I have something that I’m happy with (@borkdude also thanks to edamame), see https://gist.github.com/jeroenvandijk/080370966fadb7e65601931c3de47ed5#file-malli_inline_output-clj-L129. Here is some example output:
1: {:b :z
^------- should be an integer
2:
3: :c {:c0 2}
^------- missing required key :x
4:
5: }
^------- missing required key :a#2022-11-0909:01JHi guys! I have a question about how to check “extra constraints”. Let’s say, we have this schema:
(def CreatCommentPayload
[:map
[:comment/owner uuid?]
[:comment/text string?]
[:comment/associated-account [:vector uuid?]]])
For example, this payload is valid for create a comment:
(validate CreateCommentPayload {:comment/owner #uuid "92688fe4-9050-42b0-af5d-6d74a3b3d65b"
:comment/text "Hello"
:comment/associated-account [#uuid "864fbd90-914d-4c6d-b1b9-415276f7dda8"]}
In fact, this is not enough validation. There is some other possible validations:
• comment/owner should exist in the database.
• Each account in comment/associated-account should exist in the database.
There are several ways to check this two other facts:
1- Custom schema
Like:
(def CreateCommentPayload
[:map
[:comment/owner [:and uuid? exists?]
[:comment/text string?]
[:comment/associated-account [:and [:vector uuid?] exists?]])
exists? is a custom function to check if one or several accounts is or not in the database.
2- Constraint schema
Like:
(def CreateCommentConstraint
[:fn (fn [comment-payload]
(exists? (into (:comment/associted-account comment-payload []) (:comment/owner comment-payload))
(def CreateCommentSchema
[:and CreateCommentPayload CreateCommentConstraint])
We reuse the exists?function but here we check all the accounts with one request.
3- Don’t check constraint in schema
There will be a solution. The check of this constraints can be done in another layer of the app.
What’s your position about this case?#2022-11-0912:44Ben SlessSince the constraints you specify involve side effects, I'd say they shouldn't be in the schema#2022-11-0913:07eskosSchema is about structural validation, while you could with eg. fn schemas make them check from db all the things, it’s probably cleaner to implement that as separate validation layer. Or rely on db constraints themselves, and handle insert/update failures gracefully.#2022-11-1008:36JThanks @UK0810AQ2 and @U8SFC8HLP.#2022-11-0920:27dvingoAm I missing something obvious here?
(m/schema [:comment {:some :prop}]
{:registry (merge (m/default-schemas) {:comment :int})})
=> [:int {:some :prop}]
does not work:
(m/schema [:comment {:some :prop}]
{:registry (merge (m/default-schemas) {:comment [:map [:a :int]]})})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
:malli.core/invalid-schema {:schema [:map [:a :int]]}
#2022-11-0921:00aaron51Is a :fn schema of (.exists (io/file x)) appropriate in Malli? Given that “schema is about structural validation”? (https://clojurians.slack.com/archives/CLDK6MFMK/p1667999272184759?thread_ts=1667984513.081219&cid=CLDK6MFMK)#2022-11-1017:50CaseyIs there a function in malli already that does something like (select-keys MySchema a-map) to make a map containing only the keys defined in the schema? I don't want to define a closed schema, but I also don't want to pass unknown keys further down#2022-11-1018:06ikitommithere is an example of this in Malli readme, see https://github.com/metosin/malli#value-transformation#2022-11-1018:06Casey(select-keys (mu/keys MySchema) value) does this#2022-11-1018:07ikitommie.g. me/strip-extra-keys-transformer, it's recursive#2022-11-1018:07CaseyAh! I see, for use with (m/decode)#2022-11-1018:31CaseyThan you @U055NJ5CC!#2022-11-1513:09agigaoIn the context of coercion - can we process data before we pass it to a schema nested one level deeper?
For example:
(def file-metada
[:map
[:name string?]
[:file file-content]])
(def file-content
[:vector
[:map
[:id uuid?]
[:name string?]]])
Actually read the file and transform data before passing to file-content for validation.#2022-11-1513:25ikitommisomething I wrote.#2022-11-1603:24dumratI have been looking for something like this. I thought to myself "Surely malli must have some way to transform data - not just coerce". I will give meander a go.#2022-11-1606:25ikitommiplanning to add some helpers in Malli for this, e.g. a pre-build matcher, meander-transformer, visualization etc. Needs to play with this a bit to see what is the relevant reusable part first.#2022-11-1809:48agigaoThank you!#2022-11-1608:50Bart KleijngeldIs this expected behavior and if so, can someone point out what I'm misunderstanding?
Explicitly stating to use the default schemas registry causes an error (and actually every merge I've done so far does):
(m/validate
[:map
[:x string?]]
{:a 1}) ;; => false
(m/validate
[:map {:registry (merge (m/default-schemas))}
[:x string?]]
{:a 1}) ;; => clojure.lang.ExceptionInfo: :malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1, :max nil}
{:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :enum, :properties nil, :children nil, :min 1, :max nil}}
Shouldn't this be equivalent?#2022-11-1609:21ikitommicurrently, the property-registry doesn’t allow IntoSchema instances, just Schemas. Original idea was that you can only present serializable things. Not sure if this is a good constraint, as you can anyway have non-serializable things in schemas, e.g [:string {:gen/gen …generator-function-here…}]#2022-11-1609:22ikitommiPlease write an issue out of this.#2022-11-1609:22Bart KleijngeldI will. Thanks for the explanation#2022-11-1609:27Bart KleijngeldSee https://github.com/metosin/malli/issues/780
(If you have any title suggestion that will make this issue most findable/descriptive for you let me know)#2022-11-1612:57Bart KleijngeldI'm looking to express something like an :or schema, but instead of enumerating the arguments by hand I'd like to supply a collection of schemas.
So something like:
[:any-of schemas-coll]
instead of
[:or schema-1 schema-2 ... schema-n]
Especially nice because registries tend to be regular maps, so you can just pass those in.
Of course I can construct the :or schema above from a collection, but I would expect some nice idiom to be present.
Is there something like this present? Or should I create a new schema type for this?#2022-11-1615:01ikitommino idiom, children are currently varargs, not a single collection. But:
(into [:or] [:int :string])
; => [:or :int :string]
#2022-11-1615:03ikitommialso, there is the map ast:
(m/ast [:or :int :string])
; => {:type :or, :children [{:type :int} {:type :string}]}
(update *1 :children conj {:type :boolean})
; => {:type :or, :children [{:type :int} {:type :string} {:type :boolean}]}
(m/from-ast *1)
; => [:or :int :string :boolean]#2022-11-1615:06Bart KleijngeldThanks!#2022-11-1713:16hanDerPederIs this a bug?
(m/validate :at.least/ten 8
{:registry (merge
(m/default-schemas)
{:at.least/ten [int? {:min 10}]})}) ;; => true, but 8 < 10#2022-11-1713:18hanDerPederI would expect the props from the vector to be passed to the schema.
(m/properties
(m/schema :at.least/ten
{:registry (merge
(m/default-schemas)
{:at.least/ten [int? {:min 10}]})})) ;; => nil#2022-11-1713:28ikitommiint? is just a predicate, does not support properties, :int is a real schema. Try [:int {:min 10}] instead.#2022-11-1713:28ikitommiwould be nice to get a warning out of this, but not there yet.#2022-11-1713:30juhoteperi@U055NJ5CC predicate schemas probably support some properties, just not these#2022-11-1713:30hanDerPederaha, thanks 👍 this would imply that there is not a real schema equivalent of inst? , right? how come?#2022-11-1713:31ikitommithere is a separate ticket for making good time schemas, e.g. :time, :date , …#2022-11-1713:32ikitommibut, bumped into the missing :inst just yesterday, so, hear you.#2022-11-1713:32juhoteperi(defn -int-schema [] (-simple-schema {:type :int, :pred int?, :property-pred (-min-max-pred nil)}))
and
int? is (-simple-schema {:type 'int?, :pred int?})#2022-11-1713:33juhoteperiWe could register property-pred for predicates where it makes sense#2022-11-1713:33juhoteperi.... though I might prefer just removing property predicates completely#2022-11-1713:34hanDerPederI second that, speaking as a relatively new user coming from spec I expected int? and :int to be equivalent#2022-11-1713:35hanDerPederbut to end I would just like to say you guys but out some amazing libraries. i've learned a ton from reading your code. thanks!#2022-11-1713:38ikitommialso https://github.com/metosin/malli/issues/636#2022-11-1713:43hanDerPedershouldn't this also work?
(m/properties
(m/schema :at.least/ten
{:registry (merge
(m/default-schemas)
{:at.least/ten [:int {:doc "this number needs to be at least ten"
:custom/attrib :tranmogrify
:min 10}]})}))
ie, I expect schema properties to be returned, but is nil#2022-11-1713:52hanDerPederaha, malli.core/deref#2022-11-1716:37ikitommim/coercer and m/coerce landed on master. Runs decode + validate, throws with explain result in case of error. Have copied those fns once too many times between projects.#2022-11-1811:15ikitommihttps://github.com/metosin/malli#coercion#2022-11-1809:49lepistaneHello
I can't seem to figure out how to make humanize and explain give me reason fit for me.
So i gotta a schema
[:map
{:closed true}
[:client [:fn #function[app.validation/fn--37864]]]
[:sport [:fn #function[app.validation/fn--37866]]]
[:stream [:fn #function[app.validation/fn--37868]]]
[:timestamp [:fn #function[app.validation/fn--37870]]]]
All these are custom functions (more tight than regular predicate)
And when i send gibberish to be validated i wanted to have human readable output (in logs)
(me/humanize (m/explain schema data))
But i get
{:sport ["unknown error"], :stream ["unknown error"]}
which i don't find very useful and i would like these to be more useful.
is there anyway for these errors to be different?
solved
[:fn {:error/message "AAA"}
(fn [val] ...)))]#2022-11-1812:09ikitommithought on https://github.com/metosin/malli/pull/782?#2022-11-1908:46ikitommiIf someone has mad skills or just passion for visualizing things with javascrip/plantuml/any-open-source, would appreciate tips or help: I would like to visualize both the malli transformer chains + the meander transformations. Did the https://www.metosin.fi/blog/transforming-data-with-malli-and-meander/#visualization in the blog post with Joint+, but it’s under commercial licence, so not valid for OS. Just did a http://www.plantuml.com/plantuml/uml/fPHHxz8m4C3V-oak-_i0wYH0JBGGn1XUUT74HrFM0wfRIxgEe8HFvdVmCzckb7MXQ8nls-_krj_jRblMXcdJL2L1QOJv3XzLemi4LaD6Y_euKXnr1cS2Bq1m2hGsMiYTkzqANPirgx17IvQ0mc0rlYEJ7h5NYAQ7VeX3TtyXGV4JqpiqNf6hYWbvPzaqWhSFTTt4BW7bKJFkIPv-yLKovLzRSiJ2AEq8urnZxLZ53f7RUyzZkMTAMxlHjIApagbk4-qAQL1lMO6mEXrShL1o3FCv37xzV0bFRrv_6A_U3c6n07XQioFxgZxlhUci51BffcJ5j_RorKQLV6RmP3m4Pm3lKkEpk6G9UgxGx1M_mC2og1GooBgIuIFfILBwNvpFFOlCBcRswB1fyh3_E6sWcRIJpDrEjItw_QFqEb78jTZjJPu_3zCdn7z5b0eUeWi8PTsgXQpkVORaUE9H_5PHiZyMX73duJx0UvLshtj9axyb-SMzPOMMG-2MPvSvaMGf8PrgGaawxNXrqkylpfacd3bPekJsl_OR with PlantUML, but the layout is kinda bad and doesn’t support nested fields properly. 🧵#2022-11-1908:48ikitommi#2022-11-1908:50ikitommi#2022-11-1908:53ikitommihere’s another try with PlantUML:#2022-11-1908:57ikitommilast http://www.plantuml.com/plantuml/uml/fPH1Rk8m68NtFiMlC0lGJ6O4QAGs88fKiML3fIwHYN_0RM8Znq6qb9Fr3ZrJRT8Qfp6jgkwIv_V7txCVZ0jDbIxpZ10Kcki7cCbIfGYqW2X9Y-q_oL15i2Vm6pXB82cquc9bNvTS5NfASwof6Gs8N6FURIYzccAU47LDUkIRq_l18TvGjK9T8JAPKouQb6N9sV6ZDet7ugJO2sE_LAdpQag4-0kZR37LKbc5CgQmi5fEDmG0dkRlsgz6ajhgnr1QcZ1hL8QU2urgILE4oMvZpLoABWo7q7vv_WEjY__NtSbb1qOZWDQKRkcTl5qRj4761IQBZARtnyU_2vcnWSQThaHxBW1NeVioSaO3wZutJ98Te6qqRnGYY4uZ7SCVXEw5eB_5lAzHHAQ9qRdD-i7DVh1RMr5XUXNR4jc8PjzPrQE4CuWLNwrrF7ghOPAuwZYNEovdzY_KMtSJ6Vrim2sVEEmFgebox6dDrFlAL9MrwQgNsTbDUyzxpI8X7dZJAMm8loVXq_Gz_K02HiWO1RF_Mwy0, code expression together:#2022-11-1908:59jeroenvandijkWhat about https://mermaid-js.github.io/mermaid/#/classDiagramUnfortunately, no experience yet and not sure if it is better than PlantUML.#2022-11-1910:42ikitommiThanks, but I don’t think Mermaid supports field-level mappings.#2022-11-1912:27dumrat@U055NJ5CC I tried graphviz dot (for the first time - so this could be improved upon I guess):
https://bit.ly/3TVe9NU#2022-11-2016:56ikitommi@UC1DTFY1G that's actually quite good, bit wonky but best so far :man-bowing:#2022-11-1910:47ikitommimerged the :enum & := type-inferrer with transformers into master. So, this works now:
(m/decode
[:map
[:enum1 [:enum :kikka :kukka]]
[:enum2 [:enum 'kikka 'kukka]]
[:enum3 [:enum 1 2]]
[:enum4 [:enum 1.1 2.2]]]
{:enum1 "kikka"
:enum2 "kikka"
:enum3 "1"
:enum4 "1.1"}
(mt/string-transformer))
;{:enum1 :kikka
; :enum2 kikka
; :enum3 1
; :enum4 1.1}#2022-11-1912:57rafalwHi, I've started using malli's function intrumentation for cljs projects, it works great but on error I'm getting "Warning" message in browser's console, is it posible to make it to be an "error" instead of "warning"?#2022-11-1917:22dvingocan you post a screenshot? and how are you starting the instrumentation?
this is how they look with .cljs/start!
https://github.com/metosin/malli/blob/master/docs/clojurescript-function-instrumentation.md#errors-in-the-browser-console#2022-11-2309:25rafalwwhen its reload (shadow-cljs detects some changes) it's an warning, when page is refreshed (for example F5) it's an error#2022-11-2316:33dvingoah, i think this is due to the way shadow-cljs loads cljs code - it uses eval and prevents exceptions from being thrown as they normally do
I've found the following pattern works to get proper malli exceptions displaying:
https://github.com/matterandvoid-space/todomvc-fulcro-subscriptions/blob/766d27be316c3f2ab6a23bd8db30932ec0601a4f/src/main/space/matterandvoid/todomvc/client/dev_entry.cljs#L20#2022-11-1915:42Dannyhttps://github.com/mfikes/cljs-bean/pull/94#2022-11-1923:01aaron51How can I humanize malli.instrument/instrument! failures? With large data and a large schema, it’s really hard to tell where the problem is from the invalid-input failure#2022-11-2000:15aaron51Maybe I need to pass my own “report” function (defaults to -fail!) that throws a humanized error?#2022-11-2000:29aaron51Seems like the report function doesn’t receive parameters that can be humanized…#2022-11-2009:06ikitommi(mx/defn plus [x :- :int, y :- :int] (+ x y))
(mi/instrument! {:report (pretty/reporter)})
(plus "1" "2")#2022-11-2009:06ikitommi#2022-11-2009:06ikitommireporter is a function to create it, takes a bunch of optional options.#2022-11-2009:13ikitommiif you want to throw the humanized (instead of just printing it):
(mi/instrument! {:report (pretty/thrower)})#2022-11-2009:15ikitommi#2022-11-2016:01aaron51Awesome. Thank you!#2022-11-2009:10ikitommimalli.provider defaults now to creating real schemas (e.g. :int) instead of predicate ones (e.g. int?):
(mp/provide [{:a [1 2 3]
:b "kikka"
:c true}
{:a nil
:b "kikka"
:c "true"}])
;[:map
; [:a [:maybe [:vector :int]]]
; [:b :string]
; [:c :some]]
+ also the :some schema.#2022-11-2107:57HankstenbergHi guys, how can I repeat an element of a tuple one or more times? E.g. I'd like to have a tuple: [:tuple :int :string] and only the string part should occur one or more times. When I try :repeat or anything, the strings get wrapped into a vector. Or is there a better way to get a heterogeneous list like [12 "a" "b" "c" ...]?#2022-11-2109:00ikitommitry: [:cat :int [:+ :string]]#2022-11-2109:01HankstenbergAh, thank you very much!#2022-11-2115:42jeroenvandijkIs there a way to extend a :multi like how clojure.spec allows you to extend a multispec?#2022-11-2115:43jeroenvandijkSo far I found that all the keys of the :multi need to be in the body at definition. I would like to add entries later.#2022-11-2115:44jeroenvandijkWith a mutable registry and some hackery I can add entries, but it is not very clean#2022-11-2115:56jeroenvandijkI’m looking for something like this where I can register the dispatch type outside of the multi
(let [schema [:schema
{:registry {:x [:map [:hello :int]]}}
[:multi {:dispatch (fn [x] [:ref (:type x)])}]]]
(or (->
(m/explain schema {:type :x
:hello 1
})
(me/humanize))
:ok)) ;=> ["invalid dispatch value"]#2022-11-2116:03ikitommiYou can use references (strings or qualified keys) in multi body, like with map:
[:multi {:dispatch :type} :domain/user :domain/order]#2022-11-2116:04ikitommie.g. just reg the impls into registry reference them with the key#2022-11-2116:04jeroenvandijkAh I see now. Didn’t try qualified keys#2022-11-2116:04ikitommithere is also an example of lazy registry, could be used too: https://github.com/metosin/malli#lazy-registries#2022-11-2116:06jeroenvandijkBut it’s not possible to leave the multi without the key entries, right?#2022-11-2116:07jeroenvandijkI want to extend my schema after the first definition#2022-11-2116:08jeroenvandijkSo something like
(def CloudFormation
(m/schema
[:multi {:dispatch :Type, :lazy-refs true}]
{:registry registry}))#2022-11-2116:08ikitommiI would be easy to ad a new key e.g. :`methods` into multi, that is used for the lookups. Would that work?#2022-11-2116:09jeroenvandijkYeah that would be perfect#2022-11-2116:10jeroenvandijksomething like
.. :methods (fn [] @*my-schema)
And then I would change *my-schema somewhere else, right?#2022-11-2116:11jeroenvandijkI think this would more or less cover what multispec can do#2022-11-2116:12ikitommiPlease write a PR#2022-11-2116:12jeroenvandijkYep, thanks#2022-11-2214:50ikitommi@U0FT7SRLP I meant an Issue is enough 🙂#2022-11-2309:36jeroenvandijk@U055NJ5CC 😅 I actually tried to implement your suggestion by adding a :methods option. And it works, but after the schema is loaded it doesn't revaluate when the state/multimethod updates. Ill create an issue with an explanation and push what i have so far in a separate draft PR#2022-11-2311:06jeroenvandijk@U055NJ5CC I created an Issue https://github.com/metosin/malli/issues/786 and the :methods experiment is in https://github.com/metosin/malli/issues/787#2022-11-2209:00HankstenbergWhat is the best way to tell if a given string is a valid malli schema? As I understand it this is a two-step process... first I need to determine if it's valid edn data and in a second step I have to determine if it's a valid malli schema. There is a schema called "`:schema` " in malli.core/base-schemas , but how do I use it properly?#2022-11-2209:02ikitommivalid string like "[:enum :kikka :kukka]"?#2022-11-2209:04HankstenbergWell the idea is that the user provides a schema definition and I want to check if it's a valid malli schema, so something like "[:cat :int]", anything that can be turned into a schema with m/schema in the current context should be valid, everthing else I'd like to give proper error messages on.#2022-11-2209:05HankstenbergI'd like to humanize the output of "trying to use the input as a malli schema".#2022-11-2209:05ikitommiwithout humanizing:
(malli.edn/read-string "[:enum :kikka :kukka]")
; => [:enum :kikka :kukka]
(malli.edn/read-string "[:enumz :kikka :kukka]")
; =throws=> :malli.core/invalid-schema {:schema :enumz}#2022-11-2209:06HankstenbergYea, but is there a meta-schema for a malli schema? I thought it was maybe malli.core/base-schema/:schema#2022-11-2209:09HankstenbergIt's got to be something that me/read-string does under the hood.#2022-11-2209:49ikitommithere is no Malli Schema for Malli Schemas. There is support for it (each schema can define it's supported properties & schema for children), but no-one has had time to actually describe those.#2022-11-2209:49HankstenbergOkay good to know, thanks! 🙂#2022-11-2408:52hanDerPederwhats the point of having -update and -comp in malli.core? -comp unrolls more arities than clojure.core/comp, so I'm guessing it's a perf thing. But -update is the same as 3-arity clojure.core/update.#2022-11-2409:06ikitommiyeah, -comp is a perf thing, -update… can’t recall. looks like it’s not that useful atm.#2022-11-2414:21vemvHave you seen this error message while running a Malli-based codebase through Cloverage?
No implementation of method: :-type of protocol: #'malli.core/IntoSchema found for class: clojure.lang.PersistentList
don't know how to fix that#2022-11-2414:29vemv...I fixed it by commenting out these lines we had: ^{:type ::m/into-schema}
that pattern was sort of mindlessly copied from the Malli impl. What does it intend to do? Is it safe for us to remove?#2022-11-2416:50ikitommiit's safe to remove. It's only used for pretty printing the the reified this.#2022-11-2417:15vemvthanks! out of curiosity, do you have a sense of why that particular error is triggered?
Does overriding ^:type have some sort of code-evaluation semantic to the clj compiler?
Otherwise :type ::m/into-schema seems fairly 'inoccuous' to me, I don't see how it can trigger this issue#2022-11-2521:13dharriganBetter here I think: Got a small PR for consideration #2022-11-3014:20dharriganThank you @U055NJ5CC for merging in this small PR. I hope others may find it benefical too 🙂#2022-11-2610:49ikitommiwhat bad can happen if we add a mutable schema type? 🙈 https://github.com/metosin/malli/issues/786#issuecomment-1328020448#2022-11-2611:21pithyless> Just because we can, doesn't mean we should. 🙈
I feel if this was an issue posted on Clojure JIRA, it would require a lot more burden of proof on the poster to argue why this is really necessary, what are the practical use-cases that can't be solved otherwise, and whether the pros overcome all the cons (and future maintenance/compatibility).
For example, is breaking the validate/validator referential transparency something we want to be doing so easily? Feels like a slippery slope...
I don't have a dog in this fight, I don't really feel the pain of the original poster, but I'd argue prudence: maybe we should move this idea to an independent malli extension library until it shows its usefulness. Otherwise it will be hard to remove if it goes into malli-core and people depend on it. :)#2022-11-2613:10ikitommiI agree it should not be part of the core. I had a use-case in a project with custom reference types, where this might have been useful, not 100% sure. For the “hard to remove later”, there are some escape hatches https://github.com/metosin/malli#alpha:
> • extender API: public vars, name starts with -, e.g. malli.core/-collection-schema. Not needed with basic use cases, might evolve during the alpha, follow https://github.com/metosin/malli/blob/master/CHANGELOG.md for details
> • experimental: stuff in malli.experimental ns, code might change be moved under a separate support library, but you can always copy the old implementation to your project, so ok to use.
maybe malli.experimental.mutable :thinking_face:
(NOTE: just added the experimental documentation to README)#2022-11-2613:35ikitommioh, as the normal schemas capture forms eagerly, the whole parent-schema tree should be mutable to keep the child mutable. Not going to change all schemas to be lazy because of this.#2022-11-2613:36ikitommimutable = evil. thanks.#2022-11-2613:45ikitommi(deftest schema-test
(let [!schema (atom :int)
mutable (mem/schema (fn [options] (m/schema @!schema options)))
mutable-schema (mem/schema (fn [options] (m/schema [:map [:mutable mutable]] options)))
immutable-schema (m/schema [:map [:mutable mutable]])]
(testing "initial schema form"
(is (= [:map [:mutable :int]]
(m/form mutable-schema)
(m/form immutable-schema))))
(testing "mutating schema"
(reset! !schema [:enum "so" "mutable"]))
(testing "mutable schema changed ok"
(is (= [:map [:mutable [:enum "so" "mutable"]]]
(m/form mutable-schema))))
(testing "immutable schema captured the orginal form"
(is (= [:map [:mutable :int]]
(m/form immutable-schema)))
(testing "but workers see the mutated schema"
(is (true? (m/validate immutable-schema {:mutable "so"}))))
(testing "mutating schema again"
(reset! !schema :boolean))
(testing "form is still from the original value"
(is (= [:map [:mutable :int]] (m/form immutable-schema))))
(testing "validator is cached from the second value"
(is (true? (m/validate immutable-schema {:mutable "so"}))))
(testing "explainer is cached with the third value"
(is (nil? (m/explain immutable-schema {:mutable true}))))
(testing "conclusion"
(is (not= "mutable" "good idea"))))))#2022-11-2812:47jeroenvandijk@U055NJ5CC Thanks for trying to come up with a solution for #786. I’m reading your conclusion now 😅 Too bad#2022-11-2812:49jeroenvandijkI’ll create something internally and see how this goes. I do have use cases for this / multispec, but I can understand it might not be compatible with Malli’s design#2022-11-2813:01jeroenvandijkInspired by your example @U055NJ5CC, the following can work for me too
(defn multimethod-schema [mm]
(let [dispatch (.dispatchFn mm)]
(m/into-schema
:multi
{:dispatch dispatch}
(map (fn [[type f]] [type (f {dispatch type})]) (methods mm)))))
(defmulti mm-example :Type)
(def mm-schema
(if (= (System/getenv "ENV") "PRODUCTION")
(constantly (multimethod-schema mm-example))
(fn [] (multimethod-schema mm-example))))
(defn valid? [x]
(m/validate (mm-schema) x))
(valid? {:Type :x}) ;=> false
(defmethod mm-example :x [_]
[:map [:Type [:= :x]]])
(valid? {:Type :x}) ;=> true#2022-11-2813:02jeroenvandijkI don’t think I need much more than this, let’s see#2022-11-2813:02ikitommithat works, unless you wrap it in another schema, .e.g [:map [:mm (mm-schema)]]#2022-11-2813:03ikitommithere, :map eagerly creates the form from the current snapshot value.#2022-11-2813:04ikitommiI think it would be possible (and a good idea!) to be able to disable all caching in malli via an option, e.g. a dev-mode. would allow one to shoot him/her in the leg in new, imaginary ways. But, would also allow a “expected” repl experience, that could be turned of in normal/prod mode.#2022-11-2813:07ikitommiI think it would also clean up malli internals a lot if all caching would happen via the Cached protocol. swapping it’s impl would be trivial after that. so:
1. refactor malli-interanl caching to always use Cached
2. allow disabling caching via an option
3. welcome (optional) mutability for people who don’t have enough troubles already - or they know what they are doing 😎 #2022-11-2813:08ikitommiI can give it a shot at some point, interested in seeing if 1 would be a good idea.#2022-11-2813:11jeroenvandijkYeah interesting. To be honest I haven’t used all the features/optimizations of Malli yet. So I can’t judge how far such a change would impact things. I have been viewing Malli naively as an optimized, data oriented version of clojure.spec, but this of course has it’s own tradeoffs. I’m happy to burn myself with the above trick and report later how bad it was 😅#2022-11-2818:34jeroenvandijkBtw, I find the :multi construct very powerful. It’s also a nice tool to make the error messages very precise. I already adapted the multispec sugar to something custom to accomodate more precise errors. So maybe a generic multispec thing is not that useful after all. But I believe the option to disable caching for parts of the schema remains interesting. For now though performance is not my main concern#2022-11-2711:19CarloIs there a way to make :tuple accept lists, instead of vectors? What's the default way to spec a list?#2022-11-2711:56CarloI see that :cat takes a {:type :list} argument. I'm using this for the time being.#2022-11-2718:13ikitommiyes, :cat should accept all sequences, don’t think the :type hint does anything there. Would like to implement https://github.com/metosin/malli/issues/264 in near future, with that malli would understand the :type tags.#2022-11-2819:16mike_ananevHi, there. I’m little bit confused how to transform schema to real Swagger2 structure, from example https://github.com/metosin/malli#swagger2 ?
I want to write message specs using Malli and generate some artefacts for other teams which use Golang or Java. I’m trying to figure out how to do this?
I’ve implemented Malli Spec -> JSON Schema, but wanna use Swagger or Protobuf. The example above is incomplete and I don’t know how to generate real swagger file.#2022-11-2819:45valtteriIf you’re describing just data schemas, your swagger documentation would probably be something like
{
"swagger": "2.0"
"info": {"title": "My Title", "description": "something"},
"definitions" {
"MyObj1": <your malli->swagger-output-here>,
"MyObj2": <your malli->swagger-output-here>,
}
}#2022-11-2819:46Ben SlessFor swagger you also need data like route and method which isn't part of the schema.
Protobuf support doesn't exist yet afaik#2022-11-2819:47valtteriAFAIK swagger had slight (annoying) differences compared to JSON schema :thinking_face:#2022-11-2819:48valtteriBut anyways, as Ben said, Swagger doesn’t bring much (any) value if you’re just describing data. JSON schema should do the trick.#2022-11-2819:48valtteriThere are JSON schema libs for Java and Go#2022-11-2820:01mike_ananevOk, thank you. Got it!
It would be good to transform Malli specs to Protobuf 3 schemas cause many teams use gRPC.#2022-11-2820:36Ben SlessTell them distributed objects are bad or wait until I get to PR it to malli? 🙃#2022-11-2916:07AkizHi, I use ‘Malli’ for ‘route data validation’.
My problem is that when I validate a large structure and get an error, the output to Emacs and REPL is very slow (other IDEs don’t have this problem), I have to wait several minutes before I can work again (the error output has ~10000 lines).
I was thinking of shortening the error somehow, but I have no clear idea how. Has somebody here solved a similar problem?#2022-11-2916:18respatializedmalli.error/humanize can reduce the error size, but I'm not sure if you've already tried that or if it will suffice for your problem.#2022-11-2916:18Noah Bogarthow are you performing the validation? could you do something like (assert (m/validate X Schema) (-> X (m/explain Schema) (me/humanize)))?#2022-11-2916:20respatializeda more general purpose solution might be to defer your error logging to an async logging library like https://github.com/BrunoBonacci/mulog, which provides a ring buffer mechanism that you can hook into with custom transformation functions to output errors and logs that narrow down to the specific thing you're trying to investigate.#2022-11-2916:21AkizI forgot to tell that humanization doesnt work for me.. for unknown reason, this is my coercion in reitit-hander. It would probably help a lot.
(r.coercion.malli/create (-> r.coercion.malli/default-options (assoc :strip-extra-keys true) (update :error-keys #(conj % :humanized))))
#2022-11-2916:24Akiz@UFTRLDZEW i use timbre currently, but i would like to solve that problem on malli / emacs level first.#2022-11-2916:34ikitommiseveral minutes 🙀#2022-11-2916:34AkizOh, the error is not related to reitit coercion.
It is invalid return output value…#2022-11-2916:35Akiz@U055NJ5CC Yeah, vscode got it in second.#2022-11-2916:35ikitommido you have the pretty printing turned on?#2022-11-2916:36ikitommithe colored outputs can be slow, dunno if there is a bug in rendering, fipp is supposed to be fast.#2022-11-2916:36ikitommibut, not an emacs user (anymore), so not sure what happends there, if calva is fast with the same case.#2022-11-2916:41AkizDid you switch to vscode?
Yeah, i do, i will try to disable it, thanks.
Is there some easy way for instrumented fns to produce only humanized errors?#2022-11-2917:31Akiz• disabling pretty/reporter for instrumentation helped a lot.. closing other windows in emacs as well 😄. #2022-11-2916:58Ben SlessRegarding providing protobuf schema from malli, what do you think about defining a data representation for the protobuf spec (even as an external dependency), then defining the transformation from malli to it?
I hate protobuf but it could be useful
There are also limitations to account for, like oneOf can't be repeated, which make me pull my hair out#2022-11-3009:13dharriganI'm wondering, when invoking (mg/generate schema), if the schema contains something like [:enabled {:default true} :boolean], can the generate always use the default value, rather than flipping between true and false?#2022-11-3009:20jeroenvandijkI think you could do [:enabled {:gen/elements [true]}] Maybe I’m wrong, https://github.com/metosin/malli#value-generation#2022-11-3009:21dharriganoohh.. trying...#2022-11-3009:23dharriganYup, that works. ta! 🙂#2022-11-3020:00dharriganI'm a little confused as to where optional sits in a schema. It's not immediately clear in the documentation. Is this [:map [:name {:optional true} :string] same as [:map [:name [:string {:optional true}]]]?#2022-11-3020:18Chris O’DonnellI don't think so. Pretty sure :optional needs to be a parameter of the map key entry.#2022-11-3020:19Chris O’Donnell(The former)#2022-11-3020:19Noah Bogarti believe that the position of the {} is supposed to look like metadata on the following object#2022-11-3020:20Noah Bogartso [:map [:name :string]] and then with the optional "metadata"/params [:map [:name {:optional true} :string]]#2022-11-3020:22dharriganI see, so, [:map [:name {:optional true}]] means the name is optional. Then if I have [:map [:name {:optional true} [:string {:min 1 :max 4}]]], then it's not only optional, but I've also said that the string (if provided) must be between 1 and 4.#2022-11-3020:23dharriganso the metadata is about the proceeding "thing"#2022-11-3020:27Noah Bogartah yeah, it would be the previous thing not the following, as you've noted with the {:min 1} example#2022-11-3020:27Noah Bogart{:optional true} means that the key might not be present, whereas [:maybe X] means the value might not be present#2022-11-3020:29dharrigan👍#2022-12-0517:31escherizeThink of it like hiccup: [:div {:style "..red"} [:span ...]] the attributes maps apply to the tags immediately before them#2022-12-0517:50dharrigan👍#2022-12-0220:11Ben SlessHow about type variables for https://github.com/metosin/malli/issues/770?#2022-12-0416:02jfntnNot sure I understand, can you give an example?#2022-12-0416:17Ben Sless(map identity xs) is a classic example, identity doesn't have type any -> any, but a -> a, there's a clear relation between input and output#2022-12-0416:19jfntnThanks I see what you mean now, but isn’t that orthogonal to that particular issue?#2022-12-0417:02Ben SlessPartially. How will you write the schema for the higher order function map?#2022-12-0418:58jfntnRight, I think I see what you mean now. The instrumentation fn would unify the type variables with the actual values at runtime#2022-12-0222:56escherizein prismatic schema, you can use a class with s/defn. Is there a similar approach for mx/defn? I could check for instance in a [:fn] I suppose#2022-12-0407:25CarloA question about recursive generators and :+. Consider this recursive spec that describes a boolean formula:
(def formula
[:schema
{:registry
{::formula
[:or
:boolean
[:tuple [:enum :not] :boolean]
[:tuple [:enum :and] [:* [:ref ::formula]]]
[:tuple [:enum :or] [:* [:ref ::formula]]]]}}
[:ref ::formula]])
We can of course generate examples via:
(repeatedly 20 #(mg/generate formula))
But, if I change :* with :+ (rest in 🧵)#2022-12-0407:29CarloI now have:
(def formula
[:schema
{:registry
{::formula
[:or
:boolean
[:tuple [:enum :not] :boolean]
[:tuple [:enum :and] [:+ [:ref ::formula]]]
[:tuple [:enum :or] [:+ [:ref ::formula]]]]}}
[:ref ::formula]])
(repeatedly 20 #(mg/generate formula))
and the generator for this seems broken to me (for starters, it usually doesn't return 20 things. I suspect the reason is that it makes more difficult to generate values (as a naive sampling terminates more rarely). Is that correct?#2022-12-0410:31ikitommirecursive generators are hard, please read https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L186-L281. Ideas welcome on how to make it better.#2022-12-0602:14ambrosebsFYI I'm looking into it here: https://github.com/metosin/malli/pull/792#2022-12-0606:18ambrosebsthe issue is that recursive :+ can make generators like (gen/non-empty (gen/return ()))#2022-12-0606:43ambrosebsshould be fixed in https://github.com/metosin/malli/pull/792#2022-12-0607:20ikitommiThat was a quick resolution, thanks @U055XFK8V!#2022-12-0408:06CarloAlso, what's :malli.core/potentially-recursive-seqex supposed to represent? Can't seqexes contain recursive terms?#2022-12-0408:32CarloBy a brute force search, I saw that this problem is solved if I wrap :+ into a :schema. A more thorough explanation would be useful though#2022-12-0419:06jfntnCan someone confirm whether :schema values being explained as reified values is a bug?
malli.core
> (explain [:map [:a int?]] {:a "_"})
{:errors ({:in [:a],
:path [:a],
:schema #<
https://github.com/metosin/malli/issues/789#2022-12-0505:36Ben SlessShould and generators use bind in reverse order to allow for transformers? Sort of like specs conforming#2022-12-0518:36dharriganI wonder if anyone may have an example of using {:gen/gen ....} to generate a random email address for a malli schema?#2022-12-0518:42pithylesson my phone, but are you familiar with https://github.com/gfredericks/test.chuck? I’ve previously used string-from-regex for this. I’ve also more recently been using the faker library with malli to generate reasonable looking data if you’re not testing the extremes of the data itself (and don’t need smart shrinking)#2022-12-0518:43Ben SlessI can whip one up if you'd like
Tldr is I'd break it down to its components first, then compose their generators#2022-12-0518:45pithylesswith generators I think the question is always what’s “interesting” and in scope for the “purpose” of the output. If you’d like to test the limitations of the email string itself @UK0810AQ2 approach of composing from smaller generators is sound. #2022-12-0518:47dharriganActually#2022-12-0518:47dharriganI figured it out#2022-12-0518:47dharriganthre is an undocumented re-gen!!!#2022-12-0518:47dharriganIt pays to look at the source code#2022-12-0518:48dharrigan[:foo/email-address [:re {:gen/re-gen [email-pattern]} email-pattern]]#2022-12-0518:48Ben SlessYeah but the email regex is a monster#2022-12-0518:48dharriganYeah, it's hairy, been there before on validating email. I'll take a pragmatic approach.#2022-12-0519:50Noah Bogartis there any difference between [:id :int] and [:id int?]?#2022-12-0519:50Noah BogartI tried to search but it's kind of nebulous lol#2022-12-0519:55pithylessThey have implementation differences that leak into your code in surprising ways (if you're not aware of them). I suggest using :int by default. :)
See e.g:
https://github.com/metosin/malli/issues/636
https://github.com/metosin/malli/issues/327#2022-12-0519:59Noah Bogartah yeah, i remember seeing that a while back. that's annoying#2022-12-0616:22AbhinavThis was very interesting#2022-12-0619:05Noah BogartFound a difference between [:sequential {:max 1} schema] and [:? schema] when transformed to a json-schema: https://github.com/metosin/malli/issues/793#2022-12-0619:22Noah BogartLooks like none of the "malli.core/sequence-schemas" are handled by https://github.com/metosin/malli/blob/cda67420c215b0a8018e1adf70660826d1ba658f/src/malli/json_schema.cljc#2022-12-0621:34Noah BogartA related question: What edition of JSON Schema is Malli targeting?#2022-12-0621:16ambrosebsSome cleanup following the recent generator bugfix, including a function from schema -> code that generates generator. https://github.com/metosin/malli/pull/794#2022-12-0621:34Noah BogartA related question: What edition of JSON Schema is Malli targeting?#2022-12-1122:22Erik Colsonhi. I am trying out malli. What is the usual approach to validate a date string with malli?#2022-12-1203:29Ben SlessCoerce it first to the expected date type using a decoder#2022-12-1207:10dharriganHere's a little example that I have used (there may be other approaches)#2022-12-1207:11dharrigan(def ^:private iso8601-message "is not in ISO DATE TIME format, e.g. 2020-07-03T10:15:30+01:00 or 2020-07-03T10:15:30Z")
;; ISO_OFFSET_DATE_TIME format, e.g., 2011-12-03T10:15:30+01:00 or 2011-12-03T10:15:30Z
(defn ^:private date-time-parser
[date-time]
(try
(.parse DateTimeFormatter/ISO_OFFSET_DATE_TIME date-time)
(catch DateTimeParseException _)))
(def date-range [:map
[:from {:optional true} [:fn {:swagger/type "string" :swagger/format "date-time" :error/message iso8601-message} date-time-parser]]
[:to {:optional true} [:fn {:swagger/type "string" :swagger/format "date-time" :error/message iso8601-message} date-time-parser]]])#2022-12-1207:37Ben SlessIt's a bit draconian but I came to adopt the approach of never using a :fn schema if I can help it. Instead, all data coming into the process should go through a coercer. In reitit coercion is built in for you, otherwise just roll your own:
https://clojurians.slack.com/archives/CLDK6MFMK/p1636145457067000?thread_ts=1636057867.062800&cid=CLDK6MFMK
Then just say your date is an inst?#2022-12-1207:41eskosI used to do what @U11EL3P9U does and it works well enough IMO. While not immediately apparent, I also created https://github.com/esuomi/muotti/ partly because of the complexities related to this problem, and in fact its source has a tidbit which may also be inspiration, if nothing else: https://github.com/esuomi/muotti/blob/d6e4e3df5f70bf1d26d035b034d9306445aea423/src/main/clj/muotti/malli.clj#L181-L199#2022-12-1208:05dharrigan👍#2022-12-1211:07Erik Colsonthanks for the examples. I am going to use @U11EL3P9U’s approach on this. @UK0810AQ2 what is the reason you prefer not to use :fn ?#2022-12-1211:16Ben Sless• fn schemas are opaque
• values can't be generated
• they hide the real type you're interested in. For example, if you have a time stamp, you might want to compare it against things (such as other dates), meaning you'll have to parse it anyway. You want the real type on your hands
• Schemas specify ought, not is. you use decoding to make the best effort to bring what is to ought, then validate. You can often find an actual type which is closer to what you want than a fn#2022-12-1211:17Ben Slessalso see https://www.youtube.com/watch?v=IcgmSRJHu_8#2022-12-1211:19Ben SlessI can give you a real example - I have code which takes date over as string and needs to calculate N days ago from it, an API which returns numbers in JSON as strings. I don't want to do manual parsing. Once everything is past the gate of coercion and validation, I want my data to be the types I actually use, not what some random stressed out developer at $COMPANY crammed over a poorly thought out API#2022-12-1215:24Erik Colson@UK0810AQ2 That seems to make sense. Thanks for your explanation!#2022-12-1221:34DrLjótssonIs there a non-negligible performance penalty for sending in raw malli specs to validation functions instead of defining them with m/schema once and providing them to validation functions?#2022-12-1221:35DrLjótssonI.e.,
(def X [:int])
(m/validate x 9)
vs
(def X (m/schema [:int]))
(m/validate x 9)
#2022-12-1221:36DrLjótssonUnder the assumption that the validation function will be run multiple times of course.#2022-12-1305:25Ben SlessYou should build a validator if performance matters, can be two orders of magnitude faster#2022-12-1416:13ikitommioh, that’s a bug. fixing it in https://github.com/metosin/malli/pull/798#2022-12-1416:26ikitommimerged @US9EF3BGU#2022-12-1416:30rmxmstill I have another pickle... so
when I run explain-data, I can obtain all error messages into a sec.
however, when I define :fn it doesnt put error as "last"
as mentioned before, I will paste the real code, the gist of the problem is that map with :error/message is at different index of the vector#2022-12-1416:48rmxm(def schema
[map
[:x [:fn #(= % 2) {:error/message "not 2"}]]
[:y [:string {:min 2 :error/message "longer than 2}]])
(def payload
{:x 3
:y "a"})
(->> (mu/explain-data schema payload)
:errors
(map #(-> % :schema last :error/message)))#2022-12-1417:27ikitomminot sure what you are doing, but:
(def schema
[:map
[:x [:fn {:error/message "not 2"} #(= % 2)]]
[:y [:string {:min 2 :error/message "longer than 2"}]]])
(def payload
{:x 3
:y "a"})
(->> (mu/explain-data schema payload)
:errors
(map #(-> % :schema m/properties :error/message)))
; => ("not 2" "longer than 2")#2022-12-1417:27ikitommior you can use something like me/humanize:
(->> (m/explain schema payload)
(me/humanize))
; => {:x ["not 2"], :y ["longer than 2"]}#2022-12-1417:40rmxmgreat thanks...
well I am trying to just aggregate error strings to push them as notifications... those examples work, thanks#2022-12-1419:25escherizeI want to decode a string with an enum like so: [:enum :A :B] + "A" => :A
This doesn’t work:
(mc/decode [:enum :A :B] "A" (mtx/string-transformer))
;; => "A"
But all of these will work:
[(mc/decode [:and keyword? [:enum :A :B]] "A" (mtx/json-transformer))
(mc/decode [:and keyword? [:enum :A :B]] "A" (mtx/string-transformer))
(mc/decode [:enum {:decode/specific keyword} :A :B] "A" (mtx/transformer {:name :specific}))]
;;=> [:A :A :A]
Is there another (even better?) way to do that?#2022-12-1419:39Ben Slessdecode/string instead of specific?#2022-12-1419:53ikitomminot released, but https://github.com/metosin/malli/pull/782 last month 😎
(m/decode [:enum :A :B] "A" (mt/string-transformer))
;=> :A
#2022-12-1419:54escherizesorry @UK0810AQ2, I can’t parse that.#2022-12-1419:54escherize@U055NJ5CC Oh, neato#2022-12-1419:55escherizeIs that coming out in a release soon? (aka when?)#2022-12-1419:55ikitommiif you use deps, you can just point to the latest commit on master.#2022-12-1419:57ikitommiwill try to cut a release this year, need some work on the providers, a minor breaking change, so a minor bump to 0.10.0#2022-12-1419:57escherizeI think I’ll wait for that, then. Thanks for the info!#2022-12-1422:06escherizeIs there an analogous function to prismatic schema’s check? Here’s the docstring:
"Return nil if x matches schema; otherwise, returns a value that looks like the
'bad' parts of x with ValidationErrors at the leaves describing the failures.
If you will be checking many datums, it is much more efficient to create
a 'checker' once and call it on each of them."
#2022-12-1422:06escherizehmm, is that a key in explain?#2022-12-1422:07escherizeI have this to work with:
{:schema [:map [:x int?]],
:value {:y "3"},
:errors ({:path [:x], :in [:x], :schema [:map [:x int?]], :value nil, :type :malli.core/missing-key})}#2022-12-1422:08escherizeseems like I can reduce on errors, and assoc-in using :path or :in?#2022-12-1422:11escherizeanswering my own question here, using (me/humanize (m/errors $schema $value)) does similar thing to plumatic.schema/check ✨#2022-12-1422:11escherizeanswering my own question here, using (me/humanize (m/errors $schema $value)) does similar thing to plumatic.schema/check ✨#2022-12-1500:46rmxmanother quick question, similar to the last one, why?
(m/properties [:enum "x" "y" {:error/message "a"}]) ;; nil
(m/properties [:string {:error/message "a"}]) ;; #:error{:message "a"}#2022-12-1506:12ikitommiproperties map should be the second element in the vector syntax, not the last.#2022-12-1518:05rmxmthanks a lot, yeah silly mistake... but more and more I get more confident I will be able to express any crazy schema + have validating/schema/errors in one shape, thanks for your help#2022-12-1501:54lepistaneI have a question.
So when i am using spec to describe paramenters with reitit.ring.middleware.multipart/temp-file-part there is swagger/type which can be shown in the swagger page.
I can't seem to find a way to do this with malli (yes i swapped coercion)
I tried https://github.com/metosin/reitit/blob/master/modules/reitit-malli/src/reitit/ring/malli.cljc this breaks page so that you can't even put params anymore
:parameters {:query [:map {:json-schema {:type "file"}}
[:filename string?]
[:content-type string?]
[:size int?]
[:tempfile [:fn (partial instance? )]]]
this is the gist
Is this solvable or i am stuck with spec?
after some tryouts this worked
[:filename {:swagger/type "file"} string?]
but i am wondering am i making a mistake and this should be done some other way?
edit
https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj
provided all answers i needed, didn't see it first time i checked examples 🙂
thankful and grateful for the examples gratitude#2022-12-1503:19lepistaneI guess i didn't find answer to all questions ...
So i have custom schema that uses custom function for special case validation.
[:map {:closed true}
[:match-id [:fn #:error{:message "Invalid match-id"} #function[valid-uuid-as-str?]]]
[:date [:fn #:error{:message "Invalid date"} #function[valid-date-as-str?]]]]
But it doesn't work.
I am already using this specialized validation in other places but i was thinking about reusing it all everywhere.
is it possible that reitit parameter validation can be only primitive predicates (from clj.core and malli) ?#2022-12-1506:29ikitommithere are no restrictions on reitit for using malli, it should just work. let me test that.#2022-12-1506:36ikitommitested and it just works :thinking_face:#2022-12-1506:37ikitommiif you can do a minimalist repro, I can check what’s wrong with it.#2022-12-1511:08lepistane@ikitommi Thanks!
Here is the repo (just reused existing mali reitit example and added on top)
https://github.com/StankovicMarko/reitit/blob/custom-validation/examples/ring-malli-swagger/src/example/server.clj
You can see what i added on top: https://github.com/StankovicMarko/reitit/commit/8f6634d1f7529aebd23e6527ac738857894b0328
Main issue is that when i put values for GET plus, if they are wrong i can't make a request until i put correct value.
picture A
And for test route i can add whatever i want and i dont want that. I want to prevent it just like with 'native' types
picture B#2022-12-1511:08lepistaneWhat am i doing wrong?#2022-12-1512:33lepistaneThis was the only i managed to do it
[:map
[:date {:description "Date when match happened"
:json-schema/type "string"
:json-schema/format "date"
:json-schema/default "2022-10-10"}
[:and
[:re #"^\d{4}\-(0[1-9]|1[012])\-(0[1-9]|[12][0-9]|3[01])$"]
[:fn {:error/message "Invalid date"}
v/valid-date-as-str?]]]]
Use regex to enforce format of a string but feels wrong.
shouldn't this already be available?
https://swagger.io/docs/specification/data-models/data-types/#string#2022-12-1512:34lepistaneAm i missing something?#2022-12-1513:50lepistane@ikitommi any ideas?#2022-12-1515:39Jordan RobinsonHello, I'm in the process of porting over a codebase from spec to malli, and it's very cool! Thanks for all your work. I have a small question that may be a bit stupid so forgive my ignorance, is there a way to pick up the seed or size values from within a :gen/fmap function?#2022-12-1516:44escherizeWhat I have done, is use a deterministic transformation on the generated value in :gen/fmap#2022-12-1516:47Jordan Robinsonthat sounds like it might be along the lines of what I'm looking for, I don't suppose you have a code snippet at all?#2022-12-1516:48escherizeits not a stupid question, let me see here#2022-12-1516:50escherize(mg/generate
[:and {:gen/fmap #(str "my_" % "_thing")} string?]
{:seed 10, :size 10})#2022-12-1516:50escherizesomething like that work for you?#2022-12-1516:55Jordan Robinsonah I see, I did try something like that at first but it didn't really work for my situation, let me try and write some pseudocode for what I have
(malli/schema [:fn
{:gen/fmap
(fn [_]
(generate-thing {:seed ???}))}
(fn [thing] (validate-thing thing))])
which I'm then calling via (mg/generate schemas/thing {:seed 0})#2022-12-1516:55Jordan Robinsonso I'm not sure that would work here, does that make a bit more sense?#2022-12-1516:57escherizecan you build up a generator, and supply it in the :gen/gen property?#2022-12-1516:57escherizemeaning a clojure.test.check.generators#2022-12-1516:58Jordan Robinsonpotentially yes, but this isn't actually in test code, so I'd really only rather use clojure.test.check.generators as a last resort since that would mean putting it in the src imports which, feels a bit funky#2022-12-1516:58Jordan Robinsonif it's not a common thing what I'm doing though I guess I'd have to go down that route#2022-12-1516:58escherizeI guess it depends how hard it is to generate your thing#2022-12-1516:59Jordan Robinsonnot hard but it does take an int seed#2022-12-1516:59Jordan Robinsonand I'm not sure how to pass that through#2022-12-1516:59Jordan Robinsonsince otherwise if I call it without the seed in tests, well, it's random 😅#2022-12-1516:59escherizeis it data? it might be worth using malli to describe it#2022-12-1517:00escherizethen you will get easier generation#2022-12-1517:01Jordan Robinsonthat's a good point, I guess I'm looking for an easy way out as we already have (generate-thing)#2022-12-1517:01escherizehow about…#2022-12-1517:01escherize(mg/generate
[:and {:gen/fmap #(generate-thing %)} int?]
{:seed 10})#2022-12-1517:02escherizeO_o#2022-12-1517:02escherizeprobably not super good, since the schema is meaningless now#2022-12-1517:02Jordan Robinsonaha I wasn't sure if that would work, as if I use that to validate as well will it try and think the input should also be an int?#2022-12-1517:02Jordan Robinsonyeah#2022-12-1517:02escherizeyou can use it to generate..#2022-12-1517:02escherize(mg/generate
[:and {:gen/fmap #(generate-thing %)}
[:or int? [:fn ..check for mything..]]
{:seed 10})#2022-12-1517:03Jordan Robinsonahh that's interesting#2022-12-1517:03escherizereally weird though#2022-12-1517:03Jordan Robinsonyes very weird, 😅 I guess that's how my brain went to trying to get a seed value in at first#2022-12-1517:03escherizeyeah not sure it’s super easy#2022-12-1517:04escherizeactually.. in:
(malli/schema [:fn
{:gen/fmap
(fn [_]
(generate-thing {:seed ???}))}
(fn [thing] (validate-thing thing))])#2022-12-1517:04escherizewhat would be getting mapped over?#2022-12-1517:05Jordan Robinsonah nothing, but it does work, I ripped it from the malli test code#2022-12-1517:05Jordan Robinsonthis is the relevant example:
(is (= 42 (mg/generate [:re
{:gen/fmap (fn [_] 42)
:gen/schema :int}
#"abc"]))#2022-12-1517:06Jordan Robinson(https://github.com/metosin/malli/blob/master/test/malli/generator_test.cljc#L171)#2022-12-1517:06escherizebut you can generate values from a regex#2022-12-1517:06escherizei guess you can generate values form a :fn schema too#2022-12-1517:08Jordan Robinsonthanks very much for your help btw, I'm about to finish today but you've given me some very interesting ideas#2022-12-1517:08escherizeNice! yeah I am starting on porting a giant codebase from spec and schema (with plenty of custom macros) to malli. Fun times!#2022-12-1517:09Jordan RobinsonI'm in somewhat of a similar situation, it's fun till it isn't has been my experience so far 😅#2022-12-1517:10escherizehope your pulse doesn’t rise too much over it 😛#2022-12-2211:15Jordan Robinsonas an update on this, I ended up writing a custom generator using the clojure test check generators in the end, thanks again for all your help#2022-12-1516:38lepistaneI am unable to find a solution to what should be simple thing.
How do i convert date string to LocalDateTime object automatically?
(i am assuming this is request coercion)
This is my muuntaja instance
(def muuntaja-instance
(m/create
(-> m/default-options
(assoc-in [:formats "application/json" :decoder-opts] {:decode-key-fn csk/->kebab-case-keyword
<<???>> #(java.time.LocalDate/parse %)})
(assoc-in [:formats "application/json" :encoder-opts] {:encode-key-fn csk/->camelCaseString
:date-format "yyyy-MM-dd"})
#2022-12-1517:10Ben SlessYou have to define your own schema type for that, unfortunately, but it's pretty easy#2022-12-1517:11escherizeyesterday I wrote an x/defn macro that behaves like Schema’s s/defn. (mx/defn does not throw with incorrect :in or :out.) Is there a library or project to mirror little utilities like that in Schema or Spec?#2022-12-1517:11escherizeWhat other migration strats have you found useful?#2022-12-1517:58dvingothere is also
https://github.com/CrypticButter/snoop#2022-12-1517:59dvingomx/defn should throw when instrumentation is enabled#2022-12-1517:59escherizeit does#2022-12-1518:00escherizesnoop is cool, but i want to stick to vanilla ways of doing things#2022-12-1518:00escherizemy macro just calls instrument at defn creation time#2022-12-1518:01escherizewith a specific filter to match only the current function#2022-12-1518:09ikitommiwanted to keep mx/defn slim. thought of adding the Schema-style :always|never-validate hints too.#2022-12-1519:17escherizehow would you implement the never validate mode?#2022-12-1519:18escherizeI have a macro emitting the function with an extra {::validate! id} metadata and emitting an instrumentation filter after doing the core/defn call. so that’s a way to do the always-validate behavior#2022-12-1519:18escherizenot sure how to guard it, maybe add something into mi/-strument where it filters out on a certain fxn metadata value#2022-12-1600:06escherizerelevant bit of :always-validate https://github.com/metabase/metabase/pull/27218/files#diff-e29756aed1e2e2cbce8a26a8e4b2e1cbedfb83cd4d853cf6942455e49516e4d8R44#2022-12-1518:00Ben Sless@ikitommi taking the coercer discussion here to thread#2022-12-1518:00Ben SlessI assume the 1 arity for coercer decodes or throws, would be nice if we could have 3 arity for on success and on failure continuations, where the on-success takes the decoded value and on-failure takes the explained result#2022-12-1518:10ikitommisure, that would work as the arities woudn’t clash.#2022-12-1518:11ikitommiwant to make a PR?#2022-12-1518:12ikitommiI like the idea. control to the user.#2022-12-1518:33Ben SlessWill do#2022-12-1518:35Ben SlessSketch for more robust time schemas#2022-12-1518:35Ben Sless@ikitommi feedback / ideas before I pursue this path further?#2022-12-1518:36Ben SlessAnd thoughts regarding adding an epoch transformer?#2022-12-1606:10ikitommiLooks good. Quick comments:
• cljs + bundle size. If you want to use malli.time, it will brings the generators in, as multimethods don't get DCEd, it's a lot of extra. README defines how to check the bundle size reports. Option to have malli.time and malli.time.generators , which sucks too.
• If this is only for Clj now, then not a big issue. Could be then malli.experimental.time - when temporal/cljs solution comes, it most likely changes a bit.
• Schema types could be namespaced with time, e.g. :time/local-date, thinking of doing same for malli.util schemas.#2022-12-1607:33Ben Sless• cljs: not really sure how to approach that so I just punted and did clj only
• will move to experimental for now
• namespacing: I thought I did#2022-12-1609:01ikitomminamespace, so you did, didn’t notice, brilliant! 🙂#2022-12-1609:14juhoteperiI think it would be OK to merge and maybe even release clj-only version first#2022-12-1609:14juhoteperiThough there is some danger in that case that something in the API will be hard to implement on Cljs side#2022-12-1609:18juhoteperiI could explore using date-io as a facade to different JS date libs, so users can use whatever they want: https://github.com/dmtrKovalenko/date-io#projects#2022-12-1609:33juhoteperiAdding clj-only as experimental ns sounds good to me#2022-12-1615:28valtteriPersonally I’m a bit concerned about the longevity of date-io and would appreciate malli.time to be built upon something that will last over time. However I trust @U061V0GG2 ‘s opinion over mine 🙂 #2022-12-1615:29juhoteperiYeah the issue about Temporal support on date-io shows the the API isn't very well designed#2022-12-1615:30juhoteperiBut building our own thing to to support different libs isn't much better either#2022-12-1805:41Ben SlessBesides separating out generators code and providing acceptors for json schema, any other things to keep in mind?#2022-12-1811:53Ben SlessDraft MR is ready, your feedback is appreciated
https://github.com/metosin/malli/pull/802#2022-12-1905:57Ben Sless@ikitommi I think it's ready for review, besides adding tests the design is pretty settled.
Only thing I'm not sure about is the parsing code which exists in the core ns.#2022-12-1520:50escherizeBefore I build it myself — Is there a way to get a descirption of a schema in malli.core? e.g. [;map [:x int?]] => “A map containing: x, an integer”#2022-12-1522:47escherizeIf it did, Given a malli schema, it should return a string with a description of the shape it expects.#2022-12-1522:48escherizeAre there problems with doing that? I figure a call to m/ast, then dispatching on :type should be sufficient. wdyt?#2022-12-1605:48ikitommiyou should use m/walk + dispatch on type. Each Schema implements it's own -walk so all the children are walked correctly, see https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc on how to do this.#2022-12-1605:49ikitommibut, an interesting idea!#2022-12-1616:12escherizeCould you help me understand the benefits of using walk vs ast? Is walk more stable?#2022-12-1616:37ikitommiAll schemas implement the m/-walk. So you can walk over any schema, even those defined in user space. (m/walk schema (m/schema-walker identity)) works always. With AST, you have to know all schemas ahead of time to know how to handle those. In your use case, I guess you need to handle all schemas anyway, you can do the same with AST. Just that it breaks if someone changes the AST for a schema (each schema controls how it looks). There is a disclaimer in Malli README not to use the AST as a persistence model, could change, no no plans to break it and will avoid doing that. We are using AST walking too in projects. But, walk is always safe.#2022-12-1616:38escherizeThanks for explaining that 🙏 I am looking into walk and schema-walker now.#2022-12-1617:05escherizeHere’s a self contained example. I used cond since its easier to fit. I’ll actually use a multimethod though
(mc/walk
[:map [:x :int] [:y :string]]
(fn [schema _path children _options]
(println "-------")
(println "schema:" schema)
(println "children:" children)
(cond
(= (mc/type schema) :int) "INTEGER"
(= (mc/type schema) :string) "STRING"
(= (mc/type schema) :map) (str "map of "
(str/join "" (mapv (fn [[k _ v]] (pr-str [k v]))
children)))
:else schema)))#2022-12-1617:05escherizeI don’t think schema-walker is the way to go, since I dont want to ultimately return a schema#2022-12-1617:09escherizemaybe I can attach the stringified explaination to the schema as a property, and look for the prop on children?#2022-12-1617:09ikitommiI would copy the malli.json-schema and start from that#2022-12-1617:10ikitommiit's a postwalk, you get the transformed children to your callback, should be straightforward to implement what you want#2022-12-1606:10ikitommiLooks good. Quick comments:
• cljs + bundle size. If you want to use malli.time, it will brings the generators in, as multimethods don't get DCEd, it's a lot of extra. README defines how to check the bundle size reports. Option to have malli.time and malli.time.generators , which sucks too.
• If this is only for Clj now, then not a big issue. Could be then malli.experimental.time - when temporal/cljs solution comes, it most likely changes a bit.
• Schema types could be namespaced with time, e.g. :time/local-date, thinking of doing same for malli.util schemas.#2022-12-1613:35Thomas Moermansmall q: I sometimes see ?something arguments with the ? prefix in e.g. in Malli and other metosin code bases, is that a convention for a "maybe" (x or nil) argument?#2022-12-1613:50Ben SlessYes#2022-12-1613:50Thomas Moermancheers#2022-12-1615:44ambrosebsFWIW when I see ?schema I think "coercable to a Schema".#2022-12-1615:45ambrosebsand schema as more like "an instance of Schema"#2022-12-1615:45ambrosebsat least, that helped with my reading of malli.core#2022-12-1615:51Ben SlessOh yeah
It's a rougher sense of Maybe
Could be nil, like ?properties, could be "could be a schema, don't know yet"#2022-12-1617:08Thomas MoermanAh, yes, i see.#2022-12-1617:10Thomas MoermanThe coercion makes sense as well, especially in the context of Malli.#2022-12-1815:54respatializedbasic question: is there a mechanism for annotating/adding descriptions to basic schema types in a manner similar to annotating map keys?
Example use case: annotating tuple elements. I'd like to attach more information than just :string alone. E.g. instead of:
[:tuple :uuid :string :string]
something like:
[:tuple {:type :uuid :description "The unique identifier for thing."} {:type :string :description "The thing's annotation."} {:type :string :description "The thing's title."}]
is there an obvious way to do this that I've just totally overlooked?#2022-12-1816:08ikitommiAny schema can have properties, so:
`[:tuple
[:uuid {:description "..."}]
[:string {...}]
[:string {...}]]`#2022-12-1816:10respatializedyeah I remembered the [:string {:description "..."}] syntax not long after I posted this, thanks!
duckie#2022-12-2222:21Tiago Dall'Ocaquick question: is it normal for (mi/collect!) to be slow? I thought this macro could be at the end of every ns using defn schema metadata, but wouldn't be practical if it slowed down ns file evaluation 😕#2022-12-2222:48Tiago Dall'Ocaactually putting (do (mi/collect!) nil)) at the end of the file solved the issue#2022-12-2222:49Tiago Dall'Ocathe problem was emacs trying to print a whole bunch of text (apparently more then 1MB!)#2022-12-2222:48Tiago Dall'Ocaactually putting (do (mi/collect!) nil)) at the end of the file solved the issue#2022-12-2311:36armedHi, is there a way to apply malli function schemas (`m/=>`) to macros? Minimal example:
(def ?ErrOrOpts
[:or
[:fn {:error/message "should be Throwable"} throwable?]
[:map {:error/message "should be a map"}
[:throwable [:fn {:error/message "should be Throwable"} throwable?]]]])
(def ?LogArgs
[:function
[:=> [:cat ?Message] :nil]
[:=> [:cat ?Message ?ErrOrOpts] :nil]])
(m/=> foo ?LogArgs)
(defmacro foo
([msg] nil)
([msg err-or-opts] nil))
(comment
(foo "hello")
)
blows up with:
Invalid function arity (3):
((foo "hello") nil "hello")
....
It seems that macro call is wrapped#2022-12-2311:44armedthe workaround seems is prepend args with two :any, e.g. [:=> [:cat :any :any <real marco args>] :nil]#2022-12-2311:45armedbut error messages will contain all that extra stuff#2022-12-2313:46armedseems it is not possible to instrument macros properly, better to instrument functions which macro calls#2022-12-2706:13orestisI see there’s some datetime related work, so perhaps I just need to be patient - I noticed that inst? schemas generate java.util.Date whereas I need java.time.Instant. I can definitely provide my own generator or even converter, but I was wondering if I’m missing something. #2022-12-2706:14Ben SlessThat generator will be in my MR#2022-12-2706:15Ben SlessI'll probably get back to working on it today#2022-12-2706:15Ben SlessAnd plz review! Feedback needed#2022-12-2711:44Ben SlessAaaand the experimental time schemas MR is ready 😎#2022-12-2806:15Jungwoo KimI’m searching for datetime stuff today and meeting this thread and looked up your MR. Thank you for great work.#2022-12-2711:44Ben SlessAaaand the experimental time schemas MR is ready 😎#2022-12-2823:17ThomasCWhat’s the best way use properties with malli.core/-simple-schema?
Here’s an example:
(-> (m/-simple-schema {:type :=2
:pred #{2}
:properties {:here? 1}
:type-properties {:or-here? 2}})
m/properties)
=> nil
I would have expected the map from either :properties or :type-properties to be returned from malli.core/properties#2022-12-2905:52Ben SlessProperties go on an instance, type properties go on a simple schema definition
The constructor could be documented better, so unless you want to read the source, what are you trying to solve?#2022-12-2910:58ikitommiLike Ben said:
• :type-properties are shared for all instances, and do not contribute to form
• :properties are defined into the instance, e.g. [my-type {:kikka "kukka"}]
• there are no utilities for reading both (could be), currently the schema applications (like json-schema transformation) merges those when reading, see https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc#L172#2022-12-3007:45Lucy WangHello, how should I achieve "use default value for a field when validation fails and the value is not compatible with the schema "? e.g.
(def MySchema
[:map
[:some-int-field {:default 30} :int]
[:some-boolean-field {:default false} :boolean]])
(m/decode
MySchema
{:some-int-field "typo" :some-boolean-field "another-typo"}
what-transformer-to-put-here--i-dont-know?)
The builtin default-value-transformer would do nothing when the field is present but invalid.#2022-12-3011:11ikitommiGood Question! There is no such thing atm but doable with the current transformers:
• on :leave, validate the value, if not valid, replace with "the default". You can access the schemas (and the properties) in the transformation using the :compile hook
code would be 90% identical to the default-value-transformer , maybe could be just an option to it :thinking_face:#2022-12-3011:13ikitommigood case to try to learn how the transforformations work. Want to give it a try?#2022-12-3102:53Lucy WangAwesome! I actually ended with something exactly the same as you suggested (got some inspiration from https://github.com/metosin/malli/issues/143 )
(def fix-with-default-transformer
(mt/transformer
{:name :fix-with-default
:default-decoder
{:compile
(fn [schema _]
(if-let [{:keys [default]} (m/properties schema)]
(fn [x]
(if (m/validate schema x)
x
default))
identity))}}))
(def my-transformers (mt/transformer
mt/string-transformer
fix-with-default-transformer))
(m/decode [:int {:default 100}] "99" my-transformers)
;; => 99
(m/decode [:int {:default 100}] "what a typo!" my-transformers)
;; => 100#2022-12-3103:00Lucy WangOne more thing I want is to be able to log the map field name that has this invalid value detected and replaced, e.g.
[:map
[:number-of-adults {:default 1} [:int {:min 1}]]
[:number-of-babies {:default 0} [:int {:min 0}]]]
and when I receive invalid input {:number-of-adults "foo"} I'd like to log this down
[WARN] Got invalid input for :number-of-adults "foo", use default value '1' instead
But as I search around it looks like the path information is not available to the transformers. So I think the only way to achieve that is to use m/walk to walk the schema and add the field name to the properties, e.g. the above schema would be enriched to be:
[:map
[:number-of-adults {:default 1 :report/name :number-of-adults} [:int {:min 1}]]
[:number-of-babies {:default 0 :report/name :number-of-babies} [:int {:min 0}]]]
This shall work, albeit being a bit verbose#2022-12-3108:07ikitommiLooks good. Quick comments:
• it’s better to apply the transformation on :leave so it’s postwalk. If you just return a function from :compile, it maps to :enter phase, so it’s a prewalk. If you use nested defaults, the top-level might fail on enter, as the children are not transformed.
• returning nil from :compile is better than identity -> the transformation engine knows “there is nothing to do”
• you can create the (pure) validator just once, much faster
• so:
(defn fix-with-default-transformer []
(mt/transformer
{:name :fix-with-default
:default-decoder
{:compile
(fn [schema _]
(if-let [{:keys [default]} (m/properties schema)]
(let [valid? (m/validator schema)]
{:leave (fn [x] (if (valid? x) x default))})))}}))
• for the paths… what would you expect as a warning path from this:
(m/decode
[:map
[:x [:map
[:y [:int {:default 100}]]]]]
{:x {:y "what a typo!"}}
my-transformers)
;; => {:x {:y 100}}#2022-12-3113:31Lucy WangThanks! The validator tip is very useful.
> for the paths… what would you expect as a warning path from this:
I'll expect something like [:x :y] would be perfect.#2022-12-3017:11escherizeI might have missed this, but how do you pronounce malli? 🙂#2022-12-3017:20ikitommiGoogle pronounces it bit lazily, but close enough https://translate.google.fi/?hl=fi&sl=fi&tl=en&text=malli%0A&op=translate#2022-12-3017:22ikitommi... and, will check the describe-PR tomorrow#2022-12-3019:34escherizeNo rush, I have a copy pasted namespace in the project now. But looking forward to a version bump for a few reasons 🙂#2022-12-3108:34ikitommiThis is really good. Merged.
(is (= "one of <:dog = a map where {:x -> <integer>} | :cat = anything> dispatched by the type of animal"
(med/describe [:multi {:dispatch :type
:dispatch-description "the type of animal"}
[:dog [:map [:x :int]]]
[:cat :any]])))
(is (= "one of <:dog = a map where {:x -> <integer>} | :cat = anything> dispatched by :type"
(med/describe [:multi {:dispatch :type}
[:dog [:map [:x :int]]]
[:cat :any]])))#2023-01-0400:00escherize🙏#2023-01-0400:00escherizehappy new year 🙂#2023-01-0408:51ilmoIPA: /ˈmɑ.lːi/#2022-12-3114:37prncHello 🙂
Just checkin’—there is a coercion section in the docs https://github.com/metosin/malli#coercion but it’s something that’s not available in the latest build available on clojars https://clojars.org/metosin/malli (`0.9.2`), right?#2022-12-3117:13ikitommiNot released yet.#2023-01-0212:53Ben Slesswoohoo, time schemas merged partyparrot#2023-01-0213:40robert-stuttafordthank you!#2023-01-0711:24Jungwoo KimThank you!#2023-01-0309:11ikitommiwould this a good or a bad change? https://github.com/metosin/malli/pull/809, comments welcome.#2023-01-0313:15Ben SlessThe change is good, but the intent is no longer clear from the function's name#2023-01-0407:53ikitommiFully agree. Also, sometimes less is more. Not going to merge that.#2023-01-0407:55Ben SlessMaybe more convenient would be a variadic arity into-registry function that takes N arguments and tries to make a registry out of them#2023-01-0407:56ikitommilike composite-registy -> into-registry?#2023-01-0408:08Ben Sless🙂#2023-01-0408:08Ben Slessmaybe expose it from malli.core?#2023-01-0418:42escherizeWill me/with-spell-check ignore/passthrough if I use it on open maps or schemas? Or should I add a check to see if maps are closed before using with-spell-check?#2023-01-0513:15ikitommispell checking works on m/explain results. So, it will not see extra keys. You can close the maps with malli.util/closed-schema before calling it.#2023-01-0513:15ikitommiGood idea might be to add levels to explain, e.g. emit :warning for extra keys on open maps. then it would work for all maps.#2023-01-0513:30ikitommionly closed maps emit the "extra key" errors, which the spell checker reads, so in that sense, does not work with open maps.#2023-01-0516:57escherizeActually that’s the behavior I was hoping for. Thanks @U055NJ5CC#2023-01-0419:06escherizeAlso, just a note with malli.experimental.describe — should we find a way to unify it with :error/message or :fn/error? Right now I have a utility to assoc the same string into :describe and :error/message props, but it feels very w.e.t. (not d.r.y. + write everything twice)#2023-01-0513:17ikitommiUnification is a good idea. Issue + ideas on implementation on it most welcome.#2023-01-0513:18ikitommirelated, will work on type-tagging soon, will help to resolve these:
(defmethod accept 'float? [_ _ _ _] "float")
(defmethod accept :float [_ _ _ _] "float")
#2023-01-0608:35ikitommi@escherize does this look correct? https://github.com/metosin/malli/pull/813/commits/55975911a2bb653c10063968b33bcf35f1d21908#2023-01-0802:40DFSTThis might be a basic question, but I'm a little confused about what the numbers signify in a given path that's returned by subschemas . In the example from the readme, for example, one path that's returned is [0 :address 0 :lonlat 1]. What are the numbers for in the vec? Is there documentation somewhere that explains how to interpret a path that I missed?#2023-01-0804:37Ben SlessYou have some schema there that models an alternative. Either alt or or?#2023-01-0808:19ikitommiyes, each schema always accumulates to the :path (path in schema) even if they do not contribute to the :in (path in value). Each Schema defines it’s own meaning for :path, but by convention, numbers represent child indexes, starting from 0. e.g.
• [:tuple :int :int] has indexes 0 & 1
• [:maybe :int] has just index 0#2023-01-0808:19ikitommi(def schema [:map [:x [:tuple :int [:maybe [:map [:y [:and :int [:> 6]]]]]]]])
(mu/subschemas schemas)
;[{:path [], :in [], :schema [:map [:x [:tuple :int [:maybe [:map [:y [:and :int [:> 6]]]]]]]]}
; {:path [:x], :in [:x], :schema [:tuple :int [:maybe [:map [:y [:and :int [:> 6]]]]]]}
; {:path [:x 0], :in [:x 0], :schema :int}
; {:path [:x 1], :in [:x 1], :schema [:maybe [:map [:y [:and :int [:> 6]]]]]}
; {:path [:x 1 0], :in [:x 1], :schema [:map [:y [:and :int [:> 6]]]]}
; {:path [:x 1 0 :y], :in [:x 1 :y], :schema [:and :int [:> 6]]}
; {:path [:x 1 0 :y 0], :in [:x 1 :y], :schema :int}
; {:path [:x 1 0 :y 1], :in [:x 1 :y], :schema [:> 6]}]
(mu/get-in schema [:x 1 0 :y])
; [:and :int [:> 6]]
(mu/get-in *1 [0])
; :int#2023-01-0808:21ikitommithere are also utilities for going from between the two, mu/path->in and mu/in->paths (could be many)#2023-01-0815:43DFSTThanks! I'm trying to piece together something to derive different subscription and event paths for a re-frame db given my database schema. Given my pretty basic skillset, this is a bit of a stretch goal for me, but this is really helpful!#2023-01-0908:14ikitommi:in is the path in the value (e.g. the db), did you try that?#2023-01-1015:27ikitommiRelated to https://twitter.com/cljtogether/status/1606883291982102529, I will be spending more time on OS this year, #malli and #reitit being the first targets. We have also planned (at :metosin:) better ways to organise around our OS development, a blog post on that soon.#2023-01-1118:28escherizeThis is wonderful#2023-01-1015:52Noah Bogartthat's great news#2023-01-1113:39CarloI have a question on the workflow - I need to check my understanding; ideally I would just put a (dev/start! {:report (pretty/reporter)}) inside my user.clj so that it's executed at startup time. But, if I understand correctly that only start instrumentation for the currently loaded modules (of which there's none at the time of user.clj execution).
A workflow that works is loading some ns, then go back and dev/start! again, but is there a workaround that also keeps track of the loaded modules?#2023-01-1114:03ikitommiAlso interested in better ways to do this. You could hook it into integrant/etc workflow or make a shortcut for running that. Best would be if the would follow the new Var registration itself, tried that once, but got dirty quick. Ideas most welcome.#2023-01-1115:02CarloThank you, I agree that morally this should belong to a hook on var interning, but I did not know that was possbile at all. What would be the entry point for that?#2023-01-1116:50pithylessMy understanding was that would instrument new namespaces correctly, assuming that one used m/=> and not eg. via metadata on defn.#2023-01-1117:43Carlo@U05476190 there doesn't seem to be a difference between m/=> and metadata for this, do you have an example in mind?#2023-01-1115:16Noah BogartI'm struggling to get decoding working correctly. I want to transform a string timestamp to a clj-time instant: #clj-time/date-time "2023-01-11T15:14:48.964Z". (Yes, I know clj-time is deprecated lol). Do I have to define a custom schema for this?#2023-01-1118:31Ben SlessYes
You can look at how the time schemas are implemented#2023-01-1118:32Noah BogartOh, did those get merged? I remember seeing your PR but didn't follow closely. They've not been released yet, right?#2023-01-1118:33Ben Slessmerged but not released#2023-01-1118:33Ben Slessif you're on deps you can enjoy them right now#2023-01-1118:34Noah BogartSadly, we're using leiningen 😭#2023-01-1118:34Noah BogartI'll look at the code on master tho, thanks#2023-01-1118:30escherizeAm I doing something wrong here? I thought these would be using the same schema underneath. Maybe I was mistaken
[(m/validate [:double {:min 1}] -1.0)
(m/validate [double? {:min 1}] -1.0)]
;; => [false true]#2023-01-1118:51ikitommipredicate schemas double? don’t read any properties like real schemas :double. As this is a common cause of confusion, options to fix:
1. drop support for predicate schemas, kinda extra anyways
2. map predicate schemas into real schemas
3. finish “schema of schemas” to describe allowed properties so the latter would give a warning in… dev-mode #2023-01-1119:28escherizeYou mentioned something about not needed to define a describe multimethod for both 'integer? :integer. Is that related to 2.? It seems better to have a single schema for both. I can’t see any downsides.#2023-01-1119:29escherizeI think the only way it would break is if you had [double? {:min 1}], and it was being ignored. Which a) never happened b) is wrong anyway.#2023-01-1120:13ikitommi1️⃣ :registry property
[:schema {:registry {"ConsCell" [:maybe [:tuple :int [:ref "ConsCell"]]]}}
"ConsCell"]
2️⃣ :let schema
[:let ["ConsCell" [:maybe [:tuple :int [:ref "ConsCell"]]]]
"ConsCell"]
#2023-01-1218:26escherize:let is clearer, But :registry has the benefit of restricting shadowing / duplicate schema refs#2023-01-1209:16Lucy WangFYI, looks like metabase starts to use malli instrumentation to replace plumatic schema based instrumentation https://github.com/metabase/metabase/pull/27303/files#diff-e29756aed1e2e2cbce8a26a8e4b2e1cbedfb83cd4d853cf6942455e49516e4d8R70-R72#2023-01-1217:46escherizeThanks for noticing! 🥲 — using it to replace some clojure spec things too.#2023-01-1217:22ikitommi🥳 thank you all. A good release!#2023-01-1220:29Dallas SurewoodAnyway to make the default of a spec property to be an adjacent key? As in the default of temp-hp is max-hp#2023-01-1221:18escherizeCan you fill in what you mean a little more?#2023-01-1221:18escherizeyou probably read this already: https://github.com/metosin/malli#default-values#2023-01-1221:18Dallas SurewoodI did#2023-01-1221:21Dallas SurewoodI was hoping to use m/decode to set the defaults of a map conforming to a spec. But all of the examples show default as a constant (and the ability to use default-fn). I was hoping to map a relationship between one property and another so if that property was missing, it's default would be another property.
For instance, if temp-hp is not present, it should be set to whatever max-hp is#2023-01-1221:22Dallas SurewoodBut this is difficult to pull off.#2023-01-1221:25escherizeCan you lookup max-hp inside a function?#2023-01-1221:25escherizeor, how is it passed in?#2023-01-1221:29escherizedoes this solve it
(mc/decode [:map {:decode/player-init
(fn [{:keys [max-hp temp-hp] :as player}]
(if-not temp-hp
(assoc player :temp-hp max-hp)
player))}
[:max-hp int?]
[:temp-hp int?]]
{:max-hp 50}
(mtx/transformer {:name :player-init}))
;; => {:max-hp 50, :temp-hp 50}#2023-01-1221:31Dallas SurewoodI think that does it but I'm just looking at solutions and seeing if that's reasonable enough to write over and over. Deciding how far I can push Malli to define the shape of my maps#2023-01-1221:31Dallas SurewoodAnd whether I can make it do these things somewhat automatically#2023-01-1221:34escherizeI might write a function to generate the schema#2023-01-1221:35escherizeNot sure I can help without more info#2023-01-1221:35Dallas SurewoodI think I'm moving on from it, but thank you#2023-01-1220:30Michael Gardneris there a recommended way to wrap spec.alpha specs in Malli schemas? Of course I can do:
[:fn #(s/valid? ::foo %)]
...but then the error message is unhelpful. I can fix that too, but felt like I might be duplicating someone else's effort.#2023-01-1300:43escherizeIs anyone setting up mock functions with malli? It’s so cool!
(mx/defn eff :- string? [x :- int?]
(str x))
(def mock-eff (mg/generate (m/form (:schema (meta #'eff)))))
;; mock called with good input, returns output that matches the return schema
(mock-eff 3)
"72eSV3WQpD6d9"
;; mock throws (just like the real thing would) on bad input
(try (mock-eff "3")
(catch Exception e (ex-data e)))
;; => {:type :malli.core/invalid-input,
;; :message :malli.core/invalid-input,
;; :data {:input [:cat int?], :args ["3"], :schema [:=> [:cat int?] string?]}}#2023-01-1307:08jeroenvandijkNice trick! Thanks for sharing. Maybe also interesting to start with this implementation and complete it when you need to.#2023-01-1309:46ikitommi@escherize Malli instrumentation has a :gen option for the lazy:#2023-01-1315:29pithylessHow would you approach customizing the generators for these kind of mocks?
Would it be custom :gen at the mx/defn level? Or override the schema with custom registry at the (partial mg/generate)? Or something else?#2023-01-1315:51ikitomminot at the computer, but I think you can add :malli/gen meta to the var#2023-01-1316:25escherizeI generally have dev running when.. in.. dev. Is there a filter for that gen key?#2023-01-1316:26escherizeIt’d be nice to put a metadata on a function definition and have it autogen, then take it off recompile and no autogen. Maybe this already exists#2023-01-1314:27jeroenvandijkSo in theory you could build an application this way without filling in the details and have it somewhat working. And if it doesn’t work it would be logical errors like reusing a password, maybe other uniqueness like things, but you could get pretty far I guess. Sounds good!#2023-01-1315:24escherizeI think it’d be good for mocking an api for your hypothetical front end team#2023-01-1319:16Noah BogartWould there be any interest in having humanize say the type that's given? Instead of just {:some-num ["should be an integer"]}, it could say {:some-num ["should be an integer, given a string"]}#2023-01-1323:07escherizeyou can get something similar with https://github.com/metosin/malli/blob/master/docs/tips.md#getting-error-values-into-humanized-result#2023-01-1323:08escherizebut I agree it’s better to say the type that was given#2023-01-1408:56ikitommiyes, adding the humanized value type into the error message makes sense - as an optional :wrap impl for example.#2023-01-1319:30Dallas SurewoodWhen writing encoders/decoders for a transformer, is there a way to match by schema and not just :int or function symbols? For instance
(defn x-transformer [] (mt/transformer
{:decoders {'str-stat-schema (fn [x] (:value x))}
:encoders {'str-stat-schema (fn [x] {:value x})}}))
Was hoping that would work. Would like to make decoders to convert one schema to another#2023-01-1408:57ikitommiwhat is a 'str-stat-schema here?#2023-01-1408:59ikitommiyou can attach decode functions into any schema at the moment, e.g. have any schema control how it’s being decoded. If you want to push the control to transformer, there has to be a way to identify the schema. it can be any data pulled out of a schema.#2023-01-1409:24ikitommihere’s one way to do it using schema references (references need to be strings of qualified keywords):
(let [schema [:schema {:registry {::str-stat-schema :string}}
::str-stat-schema]
decoders {::str-stat-schema (fn [x] (:value x))}
encoders {::str-stat-schema (fn [x] {:value x})}
compiler (fn [schema _] (when (m/-ref-schema? schema) (m/-ref schema)))
transformer (mt/transformer
{:default-decoder {:compile (comp decoders compiler)}
:default-encoder {:compile (comp encoders compiler)}})
decode (m/decoder schema transformer)
encode (m/encoder schema transformer)]
((juxt identity decode (comp encode decode))
{:value "kikka"}))
; => [{:value "kikka"} "kikka" {:value "kikka"}]#2023-01-1409:27ikitommibut, it’s much easier to inject the encoders & decoders into schemas:
(def str-stat-schema
[:string {:decode/custom (fn [x] (:value x))
:encode/custom (fn [x] {:value x})}])
(-> [:map
[:x str-stat-schema]
[:y str-stat-schema]]
(m/decode
{:x {:value "kikka"}
:y {:value "kukka"}}
(mt/transformer {:name :custom})))
; => {:x "kikka", :y "kukka"}#2023-01-1409:27ikitommihope this helps#2023-01-1617:20Dallas SurewoodThis was helpful. The plan was to make easy transformations to convert from one schema to another. The first option may be best for that, but it's very verbose. I suppose I should just make my own functions that do this manually.#2023-01-1319:49dvingoI've been working on integrating malli with react-hook-form for a bit (mostly learning react-hook-form :P ), and now have something working!
https://github.com/dvingo/malli-react-hook-form
You can play with it here: https://dvingo.github.io/malli-react-hook-form/
I'm sure there may be some tricky things for more complex data shapes, but this is a promising start!#2023-01-1323:04escherizeLooks familiar! I wrote a POC for something similar a long time ago. http://escherize.com/works/data-desk#2023-01-1323:06escherizeOh, I thought you were generating this form: https://github.com/dvingo/malli-react-hook-form/blob/mainline/src/app/malli_react_hook_form/entry.cljs#L44#2023-01-1323:06escherizeit’s still really cool though#2023-01-1401:53dvingono worries! that's definitely the direction I'm heading :D (generating forms from schemas) this part was just figuring out wiring of the pieces of reusable parts, next will be outputting these from the schema#2023-01-1409:30ikitommigood to see work on malli->forms! I really would like have that, have few non-complete prototypes and most likely no time to finish.#2023-01-1411:31wcalderipeI'll share a fresh thread from #clojurescript here in an attempt to connect people with similar issues
https://clojurians.slack.com/archives/C03S1L9DN/p1673488701932789#2023-01-1415:07dvingojust updated to generate the form inputs from the schema
https://github.com/dvingo/malli-react-hook-form/blob/8f2076cc27d23c96795c45f825c474e544f5d815/src/app/malli_react_hook_form/entry.cljs#L71#2023-01-1723:53escherizeI uploaded the lastest version of my version lately too:
https://escherize.com/works/data-desk/
I think to do it right requires using a similar strategy to malli/experimental/describe.cljc where there’s a multimethod over the schema types.
Might be cool to do using htmx, too.#2023-01-1801:45dvingonice! do you have the source published or no?
And yep for sure, will need to deal with all the built in schemas plus nested/refs#2023-01-1415:07dvingojust updated to generate the form inputs from the schema
https://github.com/dvingo/malli-react-hook-form/blob/8f2076cc27d23c96795c45f825c474e544f5d815/src/app/malli_react_hook_form/entry.cljs#L71#2023-01-1517:48pithyless> Malli requires Clojure 1.10+ and is tested against 1.10 and 1.11.
There was an error reported on #C8V0BQ0M6 about a broken build, because of :as-alias being used in malli.generator (which is 1.11 only). Is the README out of date?#2023-01-1518:04ikitomminot intentional, will fix#2023-01-1606:08ikitommishould be fixed now: https://github.com/metosin/malli/pull/816#2023-01-1606:09ikitommithanks for pointing out, had missed that!#2023-01-1606:43pithylessThanks for insta-fixing it! gratitude#2023-01-1608:53moeI'm working on my first schema — what's the most concise way to specify that a value is a literal keyword (i.e. not any keyword, but a specific one)#2023-01-1608:57robert-stuttaford[:= :malli/is-awesome]#2023-01-1608:57moeHey Rob! Thanks!#2023-01-1608:57robert-stuttafordomg Moe! freakin long time dude!#2023-01-1608:58moefor real#2023-01-1608:58moemysterious ways#2023-01-1608:58robert-stuttafordglad you found The Way, as i did#2023-01-1608:58robert-stuttafordwe got up to no good back in the old days 😂#2023-01-1609:00moeI was hacking on Scheme even back then but there wasn't really anything plausible I could talk about using for work#2023-01-1609:01robert-stuttafordyeah it's come a loooong way#2023-01-1609:03moewe're still pariahs, outnumbered by COBOL developers 😕#2023-01-1610:25robert-stuttafordhaha#2023-01-1719:40escherizeWouldn’t it be nice to have a kondo hook for mx/defn?
I think using https://github.com/metosin/malli/blob/master/resources/clj-kondo/clj-kondo.exports/metosin/malli/config.edn#L1 is causing problems.
I was looking into the issue mentioned in https://github.com/metabase/metabase/pull/27647/files#diff-b30ee307d9c92516f1f5b912532eb62cb13c00cc267ed51df7953e9c97fb6ae8R11-R13, and started https://clojurians.slack.com/archives/CHY97NXE2/p1673983245295079#clj-kondo.#2023-01-1820:41escherizerelated: https://clojurians.slack.com/archives/CHY97NXE2/p1641752397035400#2023-01-1800:47Franco GasperinoGood afternoon. I ran into an unexpected behavior with default values, and was hoping someone could point me in the correct direction.#2023-01-1800:48Franco Gasperino(require '[malli.core])
(require '[malli.util])
(require '[malli.transform])
(def italian-kids #{"Paul"})
(def name-opts {:min 1 :max 16})
(def decode #(malli.core/decode %1 %2 {} malli.transform/default-value-transformer))
(def children-schema
[:map
[:children
[:set {:min 1 :default italian-kids}
[:string name-opts]]]])
(decode children-schema {})
;; => {:children #{"Paul"}}
(def family-schema
[:map
[:dad [:string (assoc name-opts :default "Paul")]]
[:mom [:string (assoc name-opts :default "Maria")]]
[:children children-schema]])
(decode family-schema {})
;; => {:dad "Paul", :mom "Maria"}
(decode family-schema {:children {}})
;; => {:children {:children #{"Paul"}}, :dad "Paul", :mom "Maria"}
#2023-01-1800:49Franco Gasperinohow can i configure the second call to decode with an empty map to behave as the third call#2023-01-1815:48Franco GasperinoThe issue seems to be the nesting of :map schema definitions (in this case) when applying the default-value-transformer.#2023-01-1816:55escherizeCan’t check now, but I think you’d need a default value on the :map in children-schema#2023-01-1816:56escherize(def children-schema
[:map <-- add default here ?
[:children
[:set {:min 1 :default italian-kids}
[:string name-opts]]]])#2023-01-1818:26Franco GasperinoIll experiment a bit. Another option may be to leverage :default-fn. Curious if there is something more straitforward that I overlooked.#2023-01-2017:28Franco Gasperinoanswering my own question - setting a :default {} onto the :children key in the family-schema will invoke the desired behavior.#2023-01-2115:31ikitommi~That sounds like a bug, please write an issue~#2023-01-2115:33ikitommiread the example wrong, not a bug.#2023-01-1818:17Brett RowberryIf I have a fn that returns a promise with an int inside. How might I best express a promise<int> in Malli?#2023-01-1819:34valtteriAt least it can be done with :fn
(def schema [:fn (fn [x] (and (realized? x) (int? (deref x))))])
(def my-promise (promise))
(m/validate schema my-promise)
;; => false
(deliver my-promise 1)
(m/validate schema my-promise)
;; => true
#2023-01-1819:34valtteriI don’t know if there’s a better way :thinking_face:#2023-01-1819:39valtteriNOTE: this only makes sense if you know that the promise should be realized already. Another option is to omit the realized? check which will make the schema block until the promise is delivered. This may lead to weird things though.#2023-01-1819:41Brett RowberryThis looks better than what I was thinking.#2023-01-1819:43Brett RowberryThe payload isn’t actually an int, I just thought that would be easier to talk about.#2023-01-1819:44valtteriYeah, promise is just a box that can hold anything. It doesn’t really care what goes in it#2023-01-1819:44Brett RowberryIt might be nice to have a wrapper schema for promise like vector and map have.#2023-01-1819:46valtteriI think schemas are most useful on values… and promises are kinda boxes that maybe will some day contain a value. To me it’s kinda hard to think schemas on things where the outcome changes over time :thinking_face:#2023-01-1819:47valtteriMy 🧠 is limited though! Interested to hear more about your use-case if you would like to share? 🙂#2023-01-1819:49Brett RowberryI have an HTTP API that uses Reitit and Malli. I’d like to make a Clojure client (not all clients are Clojure) for it that takes advantage of the same schemas.#2023-01-1819:50Brett RowberryA synchronous client function is a solved problem. The async one is where it gets weird.#2023-01-1819:51valtteriCould malli validation to be deferred to the point in time when the value is available?#2023-01-1819:51Brett RowberryIt definitely could because I’m not even doing manual validation or instrumentation in the client, it’s just for documentation.#2023-01-1819:52valtteriOr do you want to use malli to tell “this is a function that returns a promise and the time the promise is resolved it will contain an int”#2023-01-1819:52valtteriAh, I see#2023-01-1819:52Brett RowberryI trust the service to do what the schemas say since I wrote it, and the service and client will be in the same repo#2023-01-1819:53Brett Rowberry> Or do you want to use malli to tell “this is a function that returns a promise and the time the promise is resolved it will contain an int”
Yeah, that would be cool, but maybe not necessary.#2023-01-1819:57valtteriNeed to think this a little more. Currently I think that personally I’d be happy if the documentation said something like this
> Returns a promise that, once realized, will contain a [:map [:x int?]]
#2023-01-1820:47Brett RowberryI guess promise is more like maybe as a container - holds a single thing #2023-01-1820:48Brett RowberryExcept if the thing throws!#2023-01-1821:03escherizeThis is probably naive, but I think i’d do it like so:
(def schema [:fn (fn [x] (and (realized? x) (int? (deref x))))])
(def my-promise (promise))
(defn promise-getter [p schema]
(let [pv @p]
(if (mc/validate schema pv)
pv
(throw (ex-info "promise schema mismatch" (mc/explain schema pv))))))
(mc/validate schema my-promise)
;; => false
(deliver my-promise 1)
(try (promise-getter my-promise :string)
(catch Exception e (ex-data e)))
;;=> {:schema :string, :value 1, :errors ({:path [], :in [], :schema :string, :value 1})}
(promise-getter my-promise :int)
;; => 1#2023-01-2115:22ikitommiIf clj-kondo could track over promises, hinting a return value of type promise of [:map [:x int?]] could make sense as you would get static checking on how the value is accessed, e.g. needs to be unboxed first. Did not check if clj-kondo supports IDerefs.#2023-01-2115:24ikitomminot suggesting it’s a good idea, but I would:
(import '(clojure.lang IPending IDeref))
(def Promise
(m/-simple-schema
(fn [_ [s]]
(let [valid? (m/validator s)]
{:type :promise
:min 1
:max 1
:pred (fn [x] (and (instance? IPending x)
(instance? IDeref x)
(if (realized? x)
(valid? (deref x))
true)))}))))
(def PromiseOfAMap [Promise [:map [:x :int]]])
(defn delivered [x] (deliver (promise) x))
(m/validate PromiseOfAMap {:x 1})
; => false
(m/validate PromiseOfAMap (delivered {:x 1}))
; => true
(m/validate PromiseOfAMap (delivered "invalid"))
; => false
(m/form PromiseOfAMap)
; => [:promise [:map [:x :int]]]#2023-01-1821:46steveb8nQuestion: is it possible to use a custom registry with a function schema? I can’t find an example of that#2023-01-1822:58steveb8nok, I figured out a way to do it…#2023-01-1822:58steveb8n(m/=> flow-viz [:=> [:cat (m/schema ::flow {:registry (schemas)})]
:string])
(defn flow-viz
[flow]
"hi")
#2023-01-1822:58steveb8ncurious if there’s a data only way to do this e.g. with :schema#2023-01-1822:58steveb8nI couldn’t make that work#2023-01-1916:03ikitommiI think the schemas are eagerly evaluated with m/=>. I would use a custom global registry.#2023-01-1920:24steveb8nthanks. I’ll look at a custom global registry, didn’t know about that option#2023-01-1921:35steveb8n@U055NJ5CC fyi there’s a typo in the dev instrumentation docs….#2023-01-1921:35steveb8n(plus 6)
; => 7
#2023-01-1921:35steveb8nshould be plus1#2023-01-2011:07eskosFinally took the time to release this one properly. I’d say I’m about 17% ashamed of it, which to me means it’s like the perfect time to throw it out there for people to scrutinize 🙂
From Malli’s point of view Muotti is a somewhat fancy value transformer. It’s originally made to fill a need I had and works well enough for me that I want to believe it’s useful for others as well 🤞#2023-01-2015:06ikitommihttps://github.com/metosin/malli/commit/92c64b4030102404b0e822cc43485f1de182e201 link to malli README 👍#2023-01-2016:21escherizeThis is slick — hope I find a good reason to pull it in soon#2023-01-2016:30ikitommi@U8SFC8HLP related to "can provide default values for nils", does it have benefits over using mt/default-value-transformer?#2023-01-2016:59eskosProbably not practical, since it does more than mt/default-value-transformer internally.#2023-01-2017:00eskosLets consider it another flavor, maybe one doesn't want to compose transformers, or maybe defaulting to different value in different steps makes sense...probably not, but it's ambiguous and was easy to implement originally so it's there 🙂#2023-01-2309:54mkvlrhey 👋 thanks a lot for malli, it’s very useful. We’re looking at unsing function inline schemas in our codebase and wondering about a few things:
1. what should I expect from the clj-kondo integration? Should I expect and error given the following spec & call
(mx/defn transact! [{:as system :keys [datomic]} :- [:map [:datomic some?]] tx-data :- sequential?]
,,,)
(transact! {} [])
2. is there a way to assert only the presense of certain keys in a map, without the some? above? Can I maybe even https://cljdoc.org/d/metosin/malli/0.10.1/doc/readme?q=keys#destructuring?
3. when do I need to call mi/instrument!? It seems I need to call it again after I change a defn with inline function schemas, is that right?#2023-01-2309:55borkdude@U5H74UNSF 1. Try running ()#2023-01-2309:58mkvlr@U04V15CAJ that’s for the clj-kondo integration?#2023-01-2309:59borkdudealso for automatic instrumentation#2023-01-2309:59mkvlrsee the docstring now but not seeing an effect yet#2023-01-2310:00borkdudehttps://github.com/metosin/malli/blob/cfbc8eaa05bedb1e7e7b132d52bbf2e61b143f92/src/malli/dev.clj#L15#2023-01-2310:00borkdudeyou need to make a few edits before it has an effect, to re-trigger clojure-lsp / clj-kondo#2023-01-2310:00mkvlror I see an output on the REPL buffer now#2023-01-2310:03borkdudeI'll try locally#2023-01-2310:04mkvlrbut calling the function with invalid args doesn’t error, only after I call (mi/instrument!) again#2023-01-2310:04borkdudeworks here:#2023-01-2310:05mkvlrvery cool#2023-01-2310:06borkdudeI only have to re-evaluate the defn and then make 1 additional edit to re-trigger linting#2023-01-2314:22ikitommire-evaluating the mx/defn should automatically re-create clj-kondo configs when dev/start! is running.#2023-01-2316:13escherize@U5H74UNSF — Do you mean that calling a function with mx/defn with the wrong args doesn’t immediately fail?#2023-01-2316:21escherizeI had that problem, and fixed it by having the instrumentation call baked into the macro: https://github.com/metabase/metabase/blob/master/src/metabase/util/malli.clj#L53-L78#2023-01-2316:22escherizethen it behaves like s/defn. I think there was talk of including somethng like this (mx/defn ^:always-check …) but I havn’t kept up with it.#2023-01-2316:25escherizeIF we are talking about adding options to mx/defn, tho.
My wish list would have:
1. “always check args and return value.”
2. It’d be cool to have a way to say “just make the shell of a function, which takes the args, checks them against the schema, and returns a generated value from the schema.” I know it’s mechanically not difficult, just needs to be hooked up. Maybe I should make an issue.#2023-01-2316:42ikitommiIssue welcome, agree that both would be good additions.#2023-01-2317:32escherizeI broke it into 2, since I’ve written #1 already. Had some open questions though.
1) https://github.com/metosin/malli/issues/823
2) https://github.com/metosin/malli/issues/824#2023-01-2317:58dvingoPut together a PR for 1) here https://github.com/metosin/malli/pull/702/files#diff-08fdf5164d5d2562ba6f3e187911a9db8086c3d06a3d048cd8045818b2d960e5R94 main motivation at that point was for better cljs DX, which isn't needed anymore, but having a form that is always instrumented is definitely useful#2023-01-2314:07geraldodevIs it possible to compare schemas ?#2023-01-2314:17ikitommidid you check malli.util/equals?#2023-01-2314:17ikitommiundocumented, not super performant, just checks the forms are equals…#2023-01-2314:19geraldodevNo, I've looked into documentation trying to find something like equals. Thank you.#2023-01-2314:31geraldodevShouldn't (mu/equals
[:map [:foo [:maybe :string]] [:bar :some]]
[:map [:bar :some] [:foo [:maybe :string]]]) be considered equal ?#2023-01-2314:40ikitommiif ordering matters, they are not equal. But I see the value in having a utility for deep-equals, ignoring order.#2023-01-2314:40ikitommisee https://github.com/metosin/malli/issues/82#2023-01-2314:41ikitommibut, no such too at the moment.#2023-01-2314:43geraldodevI was manually running mp/provide to infer the type to inject on reitit to get on the other side with some swagger generator to typescript, so a not positional equals would help.#2023-01-2316:25escherizeIF we are talking about adding options to mx/defn, tho.
My wish list would have:
1. “always check args and return value.”
2. It’d be cool to have a way to say “just make the shell of a function, which takes the args, checks them against the schema, and returns a generated value from the schema.” I know it’s mechanically not difficult, just needs to be hooked up. Maybe I should make an issue.#2023-01-2414:21Noah BogartI've introduced instrumentation at my workplace, and someone brought up the issue of the messiness of .clj-kondo/metosin/malli-types/config.edn updating between different branches/PRs and whether we should commit the file or not. We're calling (mdev/start!) in both dev and testing environments, so I believe that's generating the files fresh every time. Could we get away with .gitignoring the file and having it be generated by all users and the CI pipeline?#2023-01-2416:22escherizeWe https://github.com/metabase/metabase/blob/master/.gitignore#L87-L88 it#2023-01-2415:19Noah BogartFor those using function instrumentation in prod, do you run (mdev/start!) somewhere in your start-up sequence?#2023-01-2415:56ikitommimalli.dev ns for development use, for prod, see malli.instrument (used by malli.dev).#2023-01-2415:56Noah BogartAh I see, okay cool. Thanks#2023-01-2510:15gravWhat's the status of converting json-schemas to malli?#2023-01-2510:15gravI saw @UK0810AQ2 mentioning something about a proof-of-concept here:
https://clojurians.slack.com/archives/C053AK3F9/p1673410476605509?thread_ts=1673402053.813639&cid=C053AK3F9#2023-01-2510:17Ben Slesshttps://gist.github.com/bsless/fc0c48c2f983cf0243f0bd72c44d1d0bb
I need to flesh this out#2023-01-2510:19delaguardo404, looks like it is private gist#2023-01-2511:00grav@UK0810AQ2 Would you mind sharing what you've got?#2023-01-2514:54Ben SlessI'll fix it when I get home#2023-01-2517:50Ben Sless@U04V4KLKC @U052XLL3A try now
https://gist.github.com/bsless/fc0c48c2f983cf0243f0bd72c44d1d0b#2023-02-0119:17Ben Sless@U02CV2P4J6S here#2023-01-2512:17flowthingGiven [:map {:foo {:default :bar}}], does Malli have a API for getting the default value?#2023-01-2512:27ikitommithat’s an empty map with one property :foo :thinking_face:#2023-01-2512:28flowthingGrah, sorry, typo. 🙂#2023-01-2512:28flowthing[:map [:foo {:default :bar} keyword?]]#2023-01-2512:34juhoteperihttps://github.com/metosin/malli#value-transformation
mt/default-value-transformer#2023-01-2512:36flowthingHmm, right, I know about that, but I was wondering whether I could introspect the schema to figure out the default value for a given key. But now that you mention it, using value transformation is probably the better route. 👍 Thanks!#2023-01-2512:40juhoteperi(:default (m/properties (mu/get map-schema :foo))) might work#2023-01-2512:41flowthingI tried that earlier, but it doesn’t seem to:
(malli.core/properties (malli.util/get [:map [:foo {:default :bar} keyword?]] :foo))
;;=> nil#2023-01-2512:42juhoteperiDid you check what just mu/get returns?#2023-01-2512:42flowthingYeah, it returns keyword?, so it’s not suprise it doesn’t work. I also tried find:
(malli.util/find [:map [:foo {:default :bar} keyword?]] :foo)
;;=> [:foo {:default :bar} keyword?]
(malli.core/properties (malli.util/find [:map [:foo {:default :bar} keyword?]] :foo))
;;=> Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
;; :malli.core/invalid-schema
but that throws.#2023-01-2512:43juhoteperiOh right, and the default property is in the map schema key props#2023-01-2512:45flowthingI mean I can get to it using find, but it doesn’t feel quite right. 🙂#2023-01-2512:54ikitommiideas most welcome for a better api, currently mu/find or m/entries on the map to iterate all entries.#2023-01-2512:56flowthingGood to know, thanks. 👍 I think coercing the map to contain the default values is the right way to go in my case, though.#2023-01-2603:54escherizeshould mx/defn do type hinting automatically?#2023-01-2604:31escherizemx/defn doesn’t grab metadata off the var, just from the metadata position, unless Im missing something. It’d be nice to pass metadata into it from the name, IMO.#2023-01-2606:52ikitommi> mx/defn doesn’t grab metadata off the var, just from the metadata position, unless Im missing something. It’d be nice to pass metadata into it from the name, IMO.
I noticed that too. But, that’s how clojure works right? but I’m onboard of making mx/defn better. auto-type-hinting would require mapping from schemas to types, right?#2023-01-2606:54escherizeI have a bunch of mx/defn improvements.#2023-01-2606:55escherizeOne I just thought of is, using catn instead of cat so
[:=> [:cat :int :int] :int]
could become
[:=> [:catn [:a int] [:c int]] :int]#2023-01-2606:55escherizethe ones I have right now, are :no-throw, :gen, :seed, and :size#2023-01-2606:56escherizeI am calling mg/generate — but I think there’s a way to use -strument to do the generation#2023-01-2606:56escherizeidk which is better#2023-01-2606:57escherizefrom the function schema doc:
(def pow-gen
(m/-instrument
{:schema [:function
[:=> [:cat :int] [:int {:max 6}]]
[:=> [:cat :int :int] [:int {:max 6}]]]
:gen mg/generate}))#2023-01-2606:57ikitommiI guess with symbols, so:`[:=> [:catn ['a int] ['c int]] :int]`#2023-01-2606:57escherizesymbols!! 😍#2023-01-2606:58ikitommiyeah, the name can be anything#2023-01-2606:52escherizeYeah. Does that mapping exist already? It’s mostly trivial. ~One question is should we it let one add a custom typehint?~#2023-01-2606:54escherizeno need to allow that unless people want it.#2023-01-2606:59ikitommino such mappings yet. I have the schema-type -> type mapping in top of the backlog, e.g. int? is a :int. simplifies things like JSON Schema mappings would do it with (java/js) type hints too.#2023-01-2607:04escherizeThat is something that will be really good to have#2023-01-2610:30Karel MiarkaHello, please how to add a custom generator? Cannot figure out: [:fn {:generator-fn #?(:clj #(Date.)
:cljs #(js/Date.))}
(fn [x] (instance? #?(:clj Date
:cljs js/Date)
x))]#2023-01-2611:16ikitommisearch for :gen/gen from the README (requires a generator, not sure if plain function is valid here.#2023-01-2611:16ikitommishould there be :gen/fn :thinking_face:#2023-01-2611:27dharrigan@U04M0JEU88Z here's a little example of using a custom generator: #2023-01-2612:46Karel MiarkaThank you both for you replies. I have fortunately found clojure.core.inst? fn to be a better replacement for my case. Still not sure how to fix my previous trouble, but the answer can be probably found by studing clojure.test.check.generators and than using :gen/gen .#2023-01-2614:24Noah Bogartstyle/usage/architecture question: we're using Malli as internal schemas (instrumentation/documentation) and then also relying on the json schema transformation for input/output validation (our system has an older custom json schema builder, lol). Combined, this means our schemas are a little unwieldy and have a lot of repetition. For example, for a single POST foo endpoint, we have
• NewFoo schema
• post-foo-payload which is the json schema transformation of the NewFoo schema
• private post-foo-decoder which is standalone to get speed from m/decoder
• decode-foo-params that uses post-foo-decoder to decode the payload
• PostFooResponse schema (which is thankfully just (mu/assoc NewFood :extra-key :uuid))
• private post-foo-response-encoder which is standalone to get speed from m/encoder
• encode-post-foo-response that uses post-foo-response-encoder to do the final-step encoding/extra key stripping before calling (compojure/response resp)
• and finally post-foo-response which is the json schema transformation of the PostFooResponse schema
That's a lot and some of it comes from using Malli to generate json schema schemas, I know, but there's still a fair amount of boilerplate happening. Does anyone have ideas for how to lessen the code footprint for situations like this?#2023-01-2615:05ikitommiif that is all for a single POST, It’s a lot. There are two good ways to organize schemas:
1. Vars
a. like plumatic, (def NewFoo (m/schema …))
b. simple, usually non-qualified keys
2. Global Registry
a. like spec, description here: https://github.com/metosin/malli#mutable-registry
b. mutable = evil, qualified keys.
for serving requests over web, the boilerplate of dealing with optimized encoders & decoders can be pushed into frameworks, e.g. reitit + malli here: https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj#L57-L81.
btw, nowadays malli caches automatically validators, explainers and generators but NOT transformers. so this is almost as fast as manually creating a validator:
(def Foo (m/schema [:map [:x :int]]))
;; caches the validator on first invocation into schema instance
(m/validate Foo {:x 1})#2023-01-2615:14Noah Bogart> nowadays malli caches automatically validators, explainers and generators
Really? That's great news.
> can be pushed into frameworks
that's what our json schema is meant to handle, but switching over entirely to malli is long and hard and there's no desire to move off of compojure, so right now i'm stuck carving out a small corner of sanity in the much larger code base.#2023-01-2615:31pithyless> • private post-foo-decoder which is standalone to get speed from m/decoder
> • decode-foo-params that uses post-foo-decoder to decode the payload
I've usually created closed over functions that internally create the decoder.
(def decode-foo-params [..] (let [decoder (..)] (fn [..] (decoder ...))
#2023-01-2615:31pithylessI've also created orchestration functions that deal with the repetitions of an operation (and you can more declaratively just pass in a map of {:my-schema ,, :post-decoder-fn ,,, :etc})#2023-01-2615:37pithylessArne also had a talk about data-driven malli which I think can serve as inspiration on how to generate more of this stuff based on some initial schemas. https://www.youtube.com/watch?v=ww9yR_rbgQs#2023-01-2615:53Noah Bogartvery cool, thanks for the talk. watching it now#2023-01-2616:35Ben SlessIf you care about performance then you should consider dropping compojure, otherwise you can borrow the coercion middlewares from reitit#2023-01-2616:49Noah Bogarthah I would love to drop compojure. getting organizational buy-in is quite challenging, sadly#2023-01-2617:04Karel MiarkaHello, is there a way and example how to define Malli schema for functions with optional named arguments like this:
(defn my-named-args-fn [& {:keys [x y z]}]
(println x y z))
(my-named-args-fn :x 1 :y 2)#2023-01-2617:34ikitommino simple built-in for that, but:
(defn my-named-args-fn [& {:keys [x y z]}]
(println x y z))
(malli.destructure/infer #'my-named-args-fn)
;[:=>
; [:cat
; [:altn
; [:map [:map
; [:x {:optional true} :any]
; [:y {:optional true} :any]
; [:z {:optional true} :any]]]
; [:args [:* [:alt
; [:cat [:= :x] :any]
; [:cat [:= :y] :any]
; [:cat [:= :z] :any] [:cat :any :any]]]]]]
; :any]#2023-01-2618:09Noah Bogartshould that be an :or? maybe I don't remember how that destructuring works#2023-01-2623:39escherizeto add, if you use mx/defn on that shape, the destructuring schema will be inferred. ( I think ).#2023-01-2623:40escherizeyea.#2023-01-2623:40escherize(require '[malli.experimental :as mx])
(mx/defn my-named-args-fn [& {:keys [x y z]}]
(println x y z))
(:schema (meta #'my-named-args-fn))
;; => [:=>
;; => [:cat
;; => [:altn
;; => [:map
;; => [:map
;; => [:x {:optional true} :any]
;; => [:y {:optional true} :any]
;; => [:z {:optional true} :any]]]
;; => [:args
;; => [:* [:alt
;; => [:cat [:= :x] :any]
;; => [:cat [:= :y] :any]
;; => [:cat [:= :z] :any] [:cat :any :any]]]]]]
;; => :any]#2023-01-2623:41escherizeyou could have a function that return that shape ^#2023-01-2706:23ikitommi@UEENNMX0T :alt & :altn are the or for sequences. That should work ok.#2023-01-2706:26ikitommibtw, like with mostly everything else in malli, you can configure the dest inferring with options, “closed map with required keys”:
(md/infer #'my-named-args-fn {::md/closed-maps true, ::md/required-keys true})
;[:=>
; [:cat
; [:altn
; [:map [:map {:closed true}
; [:x :any]
; [:y :any]
; [:z :any]]]
; [:args [:* [:alt
; [:cat [:= :x] :any]
; [:cat [:= :y] :any]
; [:cat [:= :z] :any]]]]]]
; :any]#2023-01-2706:27ikitommiit would be easy to build a custom schema to cover the both cases (map or varargs), e.g.:
[:=>
[:cat
[:kvmap
[:x :any]
[:y :any]
[:z :any]]]
:any]
… that would work the same. haven’t needed so it’s not there yet.#2023-01-2708:06Karel MiarkaThanks a lot. I will play with that and would like to create a fn to simplify the declaration.#2023-01-2620:59Michael GardnerI have some namespaces that use :malli/schema function metadata, with (m.inst/collect!) as the final line in each namespace. That stopped working in 0.9.0:
Syntax error (ClassNotFoundException) compiling at (me/foo.clj:180:1).
me.foo
Was this ever intended to work?#2023-01-2718:35Michael Gardner@U055NJ5CC I don't see anything about this in the 0.9 changelog. Should I whip up a full repro? I'd thought there were others doing this, but maybe I'm the only one?#2023-01-2811:32ikitommithere should not be any breakage. Please create an issue with minimal repro.#2023-01-2811:32ikitommiworks here ok :thinking_face:
(ns demo)
(defn plus
"adds two numbers together"
{:malli/schema [:=> [:cat :int :int] :int]}
[z y]
(+ z y))
(require '[malli.instrument])
(malli.instrument/collect! {:ns 'demo})
;#{{:clj {demo {plus {:schema [:=> [:cat :int :int] :int]
; :ns demo, :name plus}}}}}#2023-02-0109:49Bart KleijngeldFor what it's worth: I'm having the same problem.
Did you find out what was the matter @U01R1SXCAUX?#2023-02-0110:02Bart KleijngeldTo be clear:
• (mi/collect!) does not work
• (mi/collect! {:ns 'demo}) does
Upon closer inspection, the 0-arity call defaults to using *ns* as namespace. In my case its value was different from what I expected during my REPL session. Maybe the same for you Michael.#2023-02-0111:20ikitommiUgh, that doesn't look right. Not using the 0-arity, it's clearly broken.#2023-02-0111:20ikitommiIssue welcome, should be easy to fix#2023-02-0112:00Bart Kleijngeldhttps://github.com/metosin/malli/issues/834#2023-02-0112:43ikitommifixed in master.#2023-02-0117:53Michael Gardnerthank you!#2023-01-2621:02Varghiz cljHi,
I'm looking for component/solution where I can transform malli schema to clerk to document the schema/spec in html format... This is just a start but ultimate need is to convert it to something I can publish in confluent wiki#2023-01-2706:35ikitommirendering schemas + also schema editors would be awesome.#2023-02-1114:35Varghiz cljThank you @U055NJ5CC#2023-01-2623:25escherizedumb question: why not use this as a malli-schema malli schema? [:fn m/schema] I guess you can’t use it to generate, but it’ll work for certain things.#2023-01-2706:31ikitommiyes, that works if you rely on global options.#2023-01-2700:02escherizeI’d like to make https://malli.io/?value=(%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D)&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D fail (since the outermost sequence is a list not a vector)
It is valid, but I want to force the sequences enforced by :* to be vectors.
what’s a good way to do that?#2023-01-2706:34ikitommicurrently https://malli.io/?value=(%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D)&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aand%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20vector%3F%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D. Allowing type hints to everything would allow [:catn {:type :vector} …] which would be better.#2023-01-2718:34escherizeI misread your answer, thanks :}#2023-01-2812:51pithylessCan I define a generator for [:map [:a :string] [:b string]] such that I can :gen/fmap the :b based on the generated value of :a ?#2023-01-2812:53pithylesseg. if using faker I can generate:
{:company-name "Acme Inc"}
but I'd like to slugify the name via some function:
{:company-name "Acme Inc.", :slug "acme"}#2023-01-2812:56ikitommiso, :gen/fmap is not good enough here?#2023-01-2812:56pithylessShould I :gen/fmap on the :map itself?#2023-01-2812:57ikitommiyes, I would do that#2023-01-2812:57pithylessThat makes sense. Thanks for the sanity check :)#2023-01-2817:05nonrecursivehey y’all, is there a way to create a map spec where all the keys are optional, but values must have one of the keys to be valid?#2023-01-2817:06nonrecursiveeg given [:map [:a {:optional true}] [:b {:optional true}]], then {:a :foo} would be valid but {:c :foo} would not#2023-01-2817:07nonrecursivei can think of [:or [:map [:a]] [:map [:b]]] but that seems clunky?#2023-01-2818:04pithylesswell, it sounds like you don’t want optional but you’re actually modeling a union type - I’d go with :multi or :or - but if that feels clunky you probably need an :and where you first spec the map and then check with custom fn that at least one of the expected keys exists.#2023-01-2818:07pithyless(You’ve still got things like :merge to help you pull out the repetitive parts of the individual unions)#2023-01-2913:53nonrecursivethank you!#2023-01-2903:58dvingofigured out a way to get clj-kondo config output persisted during dev for cljs function schemas!
https://github.com/metosin/malli/pull/829#2023-01-3018:09jasonjcknhas anyone written a generic malli schema to spec converter function i can steal? as i transition a code base over to malli, would be a helpful tool #2023-01-3018:12pithylessI remember this thread: https://clojurians.slack.com/archives/CLDK6MFMK/p1663090650431439
No idea if anything changed from that time.#2023-01-3018:15jasonjcknThanks 🙏 this looks interesting from the thread and exactly my use case https://github.com/Blasterai/malli-datomic#2023-01-3018:16emilaasaI've got this dream as well 🙂#2023-02-0323:04kennySpec->Malli 😉 https://github.com/kennyjwilli/specalli#2023-01-3023:32escherizeI’m trying to do generative testing on some functions created with mx/defn. Is there a good way to see how many things match a filter for mi/check?
(mi/check {:filters [(mi/-filter-var #{#'my/wacky-fxn})]})
I’d like to double check that I actually got a var. Ideally check would return something like {my.ns/wacky-fxn :ok}. Does this exist somewhere already?
It looks like -strument calls into :mode after the var lookup doesn’t happen. I can build a similar thing myself in that case#2023-01-3023:46escherizeI figured it out, I can pass my own mode function into -strument!#2023-01-3104:09escherizeI ended up just passing the var into this
(defn- check-fn! [fn-var & [iterations]]
(let [iterations (or iterations 5000)]
(if-let [result ((mg/function-checker (:schema (meta fn-var)) {::mg/=>iterations iterations}) @fn-var)]
result
{:pass? true :iterations iterations})))
which will throw if the var isn’t resolvable#2023-01-3113:11flowthingWhat’s the correct way to handle this?
(malli/explain [:map [:tag [:enum :foo]]] (clojure.data.xml/sexp-as-element [:foo]))
;;=> Execution error (AbstractMethodError) at clojure.data.xml.node.Element/entryAt (node.cljc:-1).
;; Method clojure/data/xml/node/Element.entryAt(Ljava/lang/Object;)Lclojure/lang/IMapEntry; is abstract
Do I need to walk the return value of sexp-as-element, transforming clojure.data.xml.node.Elements into maps before handing them to Malli, or is there a smarter way?#2023-01-3113:22flowthingSomething like this, perhaps?
(malli/explain Schema (malli/decode Schema xml (malli.transform/transformer {:name :my-custom-thing})))#2023-01-3114:13ikitommi:thinking_face: looks like the Element is not implementing clojure.lang.Associative, which malli programs against. If you could PR that into the data.xml, I should just work.#2023-01-3114:15ikitommiotherwise, you need to transform it, works, but much slower, e.g.
(m/coerce
[:map {:type 'clojure.data.xml.node.Element} [:tag [:enum :foo]]]
{:tag :foo}
custom-transformer-to-transform-the-thing)#2023-01-3114:16ikitommior:
(m/coerce
[:map {:decode/xml ...} [:tag [:enum :foo]]]
{:tag :foo}
(mt/transformer {:name :xml}))#2023-01-3117:03flowthingThanks! Yeah, that’s pretty much what I ended up with. Performance isn’t critical in this case, so this’ll do just fine. 👍#2023-02-0208:45flowthingThis solution doesn’t work with nested [:map ,,,]s, unfortunately:
(def Schema
[:map {:decode/xml (partial into {})}
[:tag [:enum :foo]]
[:content
[:+ [:map {:decode/xml (partial into {})}
[:tag [:enum :bar]]
[:content [:+ string?]]]]]])
;;=> #'user/Schema
(malli/coerce Schema
{:tag :foo :attrs {} :content [{:tag :bar :attrs {} :content ["1"]}]}
(transform/transformer {:name :xml}))
;;=> {:tag :foo, :attrs {}, :content [{:tag :bar, :attrs {}, :content ["1"]}]}
(try
(malli/coerce Schema
{:tag :foo :attrs {} :content [{:tag :BAD :attrs {} :content ["1"]}]}
(transform/transformer {:name :xml}))
(catch clojure.lang.ExceptionInfo ex
(-> ex ex-data :data :explain error/humanize)))
;;=> {:content [{:tag ["should be :bar"]}]}
(try
(malli/coerce Schema
(xml/sexp-as-element [:foo [:bar "1"]])
(transform/transformer {:name :xml}))
(catch clojure.lang.ExceptionInfo ex
(-> ex ex-data :data :explain error/humanize)))
;;=> {:tag :foo, :attrs {}, :content [{:tag :bar, :attrs {}, :content ["1"]}]}
(try
(malli/coerce Schema
(xml/sexp-as-element [:foo [:BAD "1"]])
(transform/transformer {:name :xml}))
(catch clojure.lang.ExceptionInfo ex
(-> ex ex-data :data :explain error/humanize)))
;;=> Execution error (AbstractMethodError) at clojure.data.xml.node.Element/entryAt (node.cljc:-1).
;; Method clojure/data/xml/node/Element.entryAt(Ljava/lang/Object;)Lclojure/lang/IMapEntry; is abstract
Dunno whether custom-transformer-to-transform-the-thing would solve this problem, but I’m unclear on how I’d implement that, since I can’t find any documentation or examples on how to implement custom transformers like that.#2023-02-0208:59ikitommiSomething like:
(def elements-to-maps-transformer
"transforms all Elements into maps"
(mt/transformer
{:decoders {:map #(cond->> % (instance? Element %) (into {}))}}))#2023-02-0208:59ikitommi1. apply only to :map schemas
2. run a function that checks if it’s an Element, then make it a map#2023-02-0209:03ikitommi… and you can compose, the order matters:
(mt/transformer
elements-to-maps-transformer ;; this first
(mt/json-transformer)) ;; ... so these see maps, not emelents
#2023-02-0209:07flowthingThanks, I was missing the :map bit. 👍 Still no dice, though, unfortunately: Malli only applies the transformation to the root value.#2023-02-0209:08flowthingI think the simplest solution in this case is for me to walk the thing sexp-as-element returns and turn Elements to maps before handing the value to Malli.#2023-02-0209:11ikitommioh, true. into (into {} x) is not recursive. change it to (clojure.walk/postwalk #(cond->> % (instance? Element %) (into {}))) and should work#2023-02-0209:31flowthingYep, could do that, but I don’t think that has any benefit over just transforming the value beforehand in this case.#2023-02-0209:31flowthingFWIW, https://ask.clojure.org/index.php/12625/clojure-element-doesnt-implement-clojure-associative-entryat.#2023-01-3117:41shem{:static1 "foo"
:static2 "bar"
:var-1234 "baz"
:var-9725 "euromokko"}#2023-01-3117:43shemif we have maps like these, where the static* keys are known and the numbers ending the var* keys are not known and may be any integer, is it possible to construe a schema that also covers the var* keys?#2023-01-3117:52respatializedyou could use :map-of and a predicate schema to constrain the names of the var-* keys, then :merge with the static keys#2023-01-3118:09shemright...i'll experiment with that. thanks!#2023-01-3118:56shem(def samppeli {:static1 "foo"
:static2 "bar"
:var-1234a "baz"
:var-9725 "euromokko"})
(def varskeema (m/schema [:map-of #"^var-\d+$" :string]))
(def staticskeema (m/schema [:map [:static1 :string]
[:static2 :string]]))
(def skeema (mu/merge varskeema staticskeema))#2023-01-3118:58shemthis validates ok even though it shouldn't, probably because maps are open by default. i can't seem to find the right place for {:closed true}#2023-01-3119:19respatializedTry using mu/`update-properties` on the results of mu/merge#2023-01-3120:08escherizeor mu/closed-schema on it#2023-01-3117:45shemi couldn't find pattern matching in the keys#2023-01-3120:28escherizeWe are gonna need some more info if you are trying to get help.#2023-01-3121:40dvingowith the experimental syntax is there a way to specify a unique return type per arity? I only see one in the tests/examples.
(mx/defn a-fun ;; :- [:int {:min 0}] <--- instead of this
"docstring"
([x :- [:int {:min 0}]] :- :int (inc x))
([x :- [:int {:min 0}], y :- :int] :- [:int {:min 10}] (+ x y))
([x :- [:int {:min 0}], y :- :int & zs :- [:* :int]] :- :int (apply + x y zs)))
something like that, I'm not sure where the syntax would go. With :function schema annotations this is supported, so it's just a matter of where the syntax lives and how it's transformed.#2023-01-3121:57escherizeI don’t think so. I was looking at that the other day. notice that in the tests there is no such thing.#2023-01-3121:58escherizeyou can of course use m/=> to get that behavior. but I don’t think mx/defn can#2023-01-3122:39dvingocool cool - thanks for confirming!#2023-01-3123:40escherizeBeen noodling on this one for a while. I think I have a solution, but would be nice to hear thoughts:
I want to conditionally validate fxn annotations for input/output based on the value of an atom (a switch to turn it off or on). I am thinking of 3 modes:
1. usually : validate iff @*validate-schemas.
then either of these will override *validate-schemas:
2. ^:never-validate : never validates
3. ^:always-validate : always validates
spoiler: Currently I am emitting a call to instrument!, but I am thinking that it makes sense to use -strument, and use a function in mode…
Then again it’s probably better to just require the user to specify their auto validation wants before we start defining functions.#2023-02-0116:47escherizeJust saw the PR for this! Looking forward to seeing some more improvements on mx/defn. 🆒#2023-02-0101:26geraldodev"I'm looking at the ring-malli-lite-swagger and ring-malli-swagger of reitit and the differences in the server.clj are just in the syntax. So it looks like reitit recognizes the lite syntax without configuration. It's very nice that we can use assoc/dissoc/merge in plain Clojure and reduce clutter."#2023-02-0112:37ingesolHow would you write a function schema for (defn foo [a & {:keys [b c}] ...) ?#2023-02-0112:42delaguardoYou could use malli.destructure to infer a schema.
user=> (require '[malli.destructure :as md])
nil
user=> (defn foo [a & {:keys [b c]}])
#'user/foo
user=> (md/infer #'foo)
[:=> [:cat :any [:altn [:map [:map [:b {:optional true} :any] [:c {:optional true} :any]]] [:args [:* [:alt [:cat [:= :b] :any] [:cat [:= :c] :any] [:cat :any :any]]]]]] :any]
#2023-02-0114:04ingesolGreat thanks, didn’t remember that one#2023-02-0117:17Bart KleijngeldDoes someone know if there's a way to have :malli/schema metadata on functions be rendered into documentation by Codox?#2023-02-0117:18Bart KleijngeldSo the function schema would be visible in the docs for that function. Even a plain text dump would be nice.#2023-02-0117:27Ben SlessYou could probably do it in codox by configuring which metadata it looks at#2023-02-0117:29Bart KleijngeldYeah you're probably right. And I expect whatever theme I'll be using must also be extended to show that metadata.#2023-02-0117:36escherizeIf you need to produce strings / explainations there’s malli.experimental.describe/describe too!#2023-02-0117:36escherizehttps://github.com/metosin/malli#description#2023-02-0117:40escherizeWe are using that to generate some simple docs ^#2023-02-0119:00Bart KleijngeldThat may come in handy, thanks :)#2023-02-0117:29frankhas any progress been made on this front?#2023-02-0119:00Benjaminhttps://json-schema.org/ is there tool that generates malli from json schema and would this be useful? I was doing something with google docs api in the past and it might have been mildly useful#2023-02-0119:11Noah BogartThere's https://github.com/metosin/malli/issues/54 about it, along with this https://github.com/metosin/malli/pull/211. Maybe you or someone else could pick it up and finish it? I'd also love to have this functionality but my job only allows me enough time to fix a bug here or there.#2023-02-0119:16Ben SlessThere's also a gist I shared some time ago in malli, need to dig it up#2023-02-0213:58Ben SlessConundrum: how can we make sure a user's pred implementation in custom simple schema is correct?
Is it enough that it returns truthy value, or is boolean strictly required?
The validator doc string guarantees it returns a function which returns a boolean, but it's pretty easy to violate this.
Thoughts?#2023-02-0214:12respatializedclearly, the only solution is a self-describing meta-schema for all Malli schemas 😉 #2023-02-0214:14Ben SlessAND malli dev instrumentation running in the background to catch everything#2023-02-0214:14Ben SlessThat's not a bad idea, btw#2023-02-0214:14Ben SlessA hassle, but could work#2023-02-0215:30dvingothanks to Thomas Heller, I was able to simplify the implementation of clj-kondo support in cljs https://github.com/metosin/malli/pull/833
(this supersedes my prior PR for this) this also updates the general instructions for use with cljs to use preloads which vastly simplifies dev vs release config#2023-02-0215:42ikitommiThis is brilliant!! I'll check it tomorrow 🙇#2023-02-0216:55pithylessVery cool, thanks for working on this! I'm not currently doing much CLJS work, but I remember there were some rough edges in getting the dev experience right.#2023-02-0219:53dvingofor sure! I've definitely learned a ton about shadow-cljs and the cljs compiler in general working on this. I think the big improvement since the early implementation is instrumenting at runtime (also a T Heller suggestion :) ) that happened with this PR https://github.com/metosin/malli/pull/755 it's more deterministic as you're not having to think about what is a macro and what is not#2023-02-0222:11escherizeNice!#2023-02-0216:09Ngoc KhuatHi guys, the error/fn key takes a function with 2 arities, what is the second argument for?#2023-02-0221:48ikitommioptions, https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L136-L139#2023-02-0221:38nyor.trHi, how can I make a schema which encodes a boolean to a string and vice versa?
I tried:
(def int-boolean-schema
(m/-simple-schema
(fn [_ _]
{:type :int-boolean
:type-properties {:decode/string
(fn [s]
(when-not (clojure.string/blank? s)
(get {"0" false "1" true} s)))
:encode/string
(fn [v]
(prn "val" v)
(if (true? v)
"1"
"0"))}})))
Decoding and encoding true works as expected, but encoding false returns nil without entering the :encode/string functions.
;; correct:
(m/decode [:int-boolean] "0" mt/string-transformer)
=> false
;; correct:
(m/decode [:int-boolean] "1" mt/string-transformer)
=> true
;; correct:
(m/encode [:int-boolean] true mt/string-transformer)
=> "1"
;; wrong, it should be "0":
(m/decode [:int-boolean] false mt/string-transformer)
=> nil
#2023-02-0222:41hanDerPederShouldn't this work?
(m/schema [:map {:registry {:my/int [:int {:doc "My int, it's very special"}]}}
[:key [:my/int {:foo "bar"}]]])
throws invalid schema exception
:malli.core/invalid-schema
{:type :malli.core/invalid-schema,
:message :malli.core/invalid-schema,
:data {:schema [:int {:doc "My int, it's very special"}]}}
If I remove the properties from either the registry or the map type it's fine. Maybe properties should be merged?#2023-02-0300:17dvingoI think I ran into something similar before (I didn't figure out why it fails yet):
(m/schema [:comment {:some :prop}]
{:registry (merge (m/default-schemas) {:comment :int})})
=> [:int {:some :prop}]
does not work:
(m/schema [:comment {:some :prop}]
{:registry (merge (m/default-schemas) {:comment [:map [:a :int]]})})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
:malli.core/invalid-schema {:schema [:map [:a :int]]}
#2023-02-0309:25hanDerPederI suspect stepping through malli.core/schema with these inputs will shed some light on this. I'll take a look when time permits.#2023-02-0803:09dvingoI ran into this again and looked into what's going on. have a fix here:
https://github.com/metosin/malli/pull/848#2023-02-0808:33hanDerPederNice!#2023-02-0815:31dvingoso it isn't supported to use the vector form in the registry. Will need to change strategies in our apps to handle this desired behavior#2023-02-0815:32dvingoone idea is to use :schema
(m/schema [:schema {:some :prop} :comment]
{:registry (merge (m/default-schemas) {:comment [:map [:a :int]]})})#2023-02-0819:16dvingoalso found that add a :ref works
(m/schema [:map {:registry {:my/int [:int {:doc "My int, it's very special"}]}}
[:key [:ref {:foo "bar"} :my/int]]])
#2023-02-0302:11Noah BogartCan I instrument outside code?#2023-02-0317:25escherizelike in another ns? not currently#2023-02-0303:11Noah BogartLooks like => would have to be modified to use the given symbols namespace, but otherwise works well! Maybe I’ll open a PR#2023-02-0323:10Michael Gardnerwhat's the right way to "unwrap" a :maybe in a map?
[:map [:x [:maybe :int]]] => [:map [:x :int]]
I can't quite figure out how to use malli.util/update(-in) to accomplish this. Do I need to use a walker instead?#2023-02-0408:30ikitommiJust for one specific field or traversing the schemas and changing all maybe schemas?#2023-02-0414:38ikitommiAll schemas implement the LensSchema , so you can work in same way as with clojure.core + normal data:
With clojure core:
(update {:x [:int]} :x get 0)
; => {:x :int}
With Malli:
(mu/update [:map [:x [:maybe :int]]] :x mu/get 0)
; => [:map [:x :int]
when implementing malli.util, there was a quick discussion what if we just implemented the Clojure Core protocols for Schemas, so you could use normal clojure core functions with them. So, this would work:
(update (m/schema [:map [:x [:maybe :int]]]) :x get 0)
; => [:map [:x :int]
… but it would have introduced problems when working with forms (which is just clojure data):
;; correct result when working with a schema
(update (m/schema [:maybe [:map [1 :int]]]) 0 get 1)
; => [:maybe :int]
;; wrong result when applied to form
(update [:maybe [:map [1 :int]]] 0 get 1)
;; => [nil [:map [1 :int]]]
… so decided to add dedicated helpers in malli.util for schemas#2023-02-0618:16Michael Gardnernice, thank you. I've been wrapping my schemas with m/schema for fail-fast behavior in case of invalid schemas, but it does mean modifying them requires the m.util helpers#2023-02-0414:38ikitommiAll schemas implement the LensSchema , so you can work in same way as with clojure.core + normal data:
With clojure core:
(update {:x [:int]} :x get 0)
; => {:x :int}
With Malli:
(mu/update [:map [:x [:maybe :int]]] :x mu/get 0)
; => [:map [:x :int]
when implementing malli.util, there was a quick discussion what if we just implemented the Clojure Core protocols for Schemas, so you could use normal clojure core functions with them. So, this would work:
(update (m/schema [:map [:x [:maybe :int]]]) :x get 0)
; => [:map [:x :int]
… but it would have introduced problems when working with forms (which is just clojure data):
;; correct result when working with a schema
(update (m/schema [:maybe [:map [1 :int]]]) 0 get 1)
; => [:maybe :int]
;; wrong result when applied to form
(update [:maybe [:map [1 :int]]] 0 get 1)
;; => [nil [:map [1 :int]]]
… so decided to add dedicated helpers in malli.util for schemas#2023-02-0411:29Ben Sless(defmethod accept :or [_ _ _ _] (into #{} (map accept children))) ;;??#2023-02-0414:25ikitommiinteresting, would that work with complex types like [:or [:map [:x :int]] [:map [:y :string]]]?#2023-02-0416:52Ben SlessSet of maps?#2023-02-0414:11hanDerPederthis is a bug, right?
(def t (mt/transformer
{:name :hello
:decoders {:string (constantly "HELLO")}}))
(def schema-1 [:map [:a :string] [:b :string]])
(def schema-2 [:or [:map [:a :string] [:b :string]]])
(def schema-3 [:or [:map [:a :string]]])
(def schema-4 [:map [:a :string]])
(def v {:a 1234})
(m/decode schema-1 v t) ;; {:a "HELLO"}
(m/decode schema-2 v t) ;; {:a 1234}
(m/decode schema-3 v t) ;; {:a "HELLO"}
(m/decode schema-4 v t) ;; {:a "HELLO"}
#2023-02-0414:22ikitommiIt’s a feature. :or transformer validates the result after transformation, if the result is not valid, it will ignore that branch. This could be improved, e.g.
1. take the first branch which becomes valid after the transformation (current)
2. if none is valid (new)
a. take the first branch OR
b. take the first branch where the value changed OR
c. take the branch is is “most valid” with some value distance metering
… none of these would make it bullet proof, but maybe suck less.#2023-02-0414:22ikitommiwhat do you think?#2023-02-0414:22ikitommiI think the 2a or 2b would be simple and good improvements.#2023-02-0414:23ikitommi.. and both would yield same result in your example.#2023-02-0414:33hanDerPederaha, I see.. hmm, tricky..
my case requires me to have multiple transformation passes (default values, some type coercions, etc..) before it's valid. of the cases you outlined 2b would get me closest, but I still think it would fail.
previously I tried using a multi-schema instead of or since all objects have an identity key, but ran into the issue mentioned in the FIXME comment; dispatch value is not valid until transformed#2023-02-0414:41ikitommithere is an example for the dispatch key thing with :multi on README:
:dispatch values should be decoded before actual values:
(m/decode
[:multi {:dispatch :type
:decode/string #(update % :type keyword)}
[:sized [:map [:type [:= :sized]] [:size int?]]]
[:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
{:type "human"
:name "Tiina"
:age "98"
:address {:country "finland"
:street "this is an extra key"}}
(mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
;{:type :human
; :name "Tiina"
; :address {:country :finland}}
#2023-02-0414:48hanDerPederRegarding or. Couldnt you merge all the cases and transform using that?#2023-02-0414:53hanDerPederby no means bulletproof either, but fits with the best effort mantra of transformers#2023-02-0414:55ikitommi> Couldnt you merge all the cases and transform using that?
>
could you describe how this would work?#2023-02-0414:59hanDerPedergiven or schema ?schema, do (reduce mu/merge nil (m/children ?schema)) and call transform on that#2023-02-0415:00hanDerPederand you wouldnt validate the result#2023-02-0415:02hanDerPederi do this when generating datomic pull patterns for or schemas, so i end up overfetching and clean up using strip-keys#2023-02-0509:55ikitommimu/merge overrides if the values are of different types (like normal merge):
(->> [:or
[:map [:x :int]]
[:vector :int]
:int]
(m/children)
(reduce mu/merge nil))
; => :int#2023-02-0509:56ikitommieven if they are maps, I don’t think this would work:
(->> [:or
[:map [:x :int]]
[:map [:x :string]]
[:map [:x [:maybe :uuid]]]]
(m/children)
(reduce mu/merge nil))
; => [:map [:x [:maybe :uuid]]]#2023-02-0509:57ikitommie.g. if someone passes {:x "1"} it fails on the last branch, being invalid.#2023-02-0509:57ikitommiusing me/union might be ok here:
(->> [:or
[:map [:x :int]]
[:map [:x :string]]
[:map [:x [:maybe :uuid]]]]
(m/children)
(reduce mu/union nil))
; => [:map [:x [:or [:or :int :string] [:maybe :uuid]]]]#2023-02-0509:59ikitommiwould work at least in your example:
(->> [:or [:map [:a :string] [:b :string]]]
(m/children)
(reduce mu/union nil))
; => [:map [:a :string] [:b :string]]#2023-02-0510:02ikitommihere, the order matters:
(map
(m/decoder
(->> [:or
[:map [:x :int]]
[:map [:x :string]]
[:map [:x [:maybe :uuid]]]]
(m/children)
(reduce mu/union nil))
(mt/string-transformer))
[{:x 1} {:x "1"}, {:x "kikka"}, {:x (str (random-uuid))}])
;({:x 1}
; {:x 1}
; {:x "kikka"}
; {:x "ed4e918e-2e4f-49d0-a0ee-d9805ffe6384"}) ;; already ok at the string branch#2023-02-0510:03ikitommiswapping the order:
(map
(m/decoder
(->> [:or
[:map [:x [:maybe :uuid]]]
[:map [:x :string]]
[:map [:x :int]]]
(m/children)
(reduce mu/union nil))
(mt/string-transformer))
[{:x 1} {:x "1"}, {:x "kikka"}, {:x (str (random-uuid))}])
;({:x 1}
; {:x 1}
; {:x "kikka"}
; {:x #uuid"ed4e918e-2e4f-49d0-a0ee-d9805ffe6384"})#2023-02-0510:36ikitommisummary:
1. mu/union gets you close, but need to be mindful about the order
2. would need more testing to verify that it doesn’t give wrong answers, e.g. with nested values
3. making default transformers depend on malli.utilit would mean than in CLJS, anyone using any transformations, would need to pull a lot of extra code for this, making the js bundle size much larger by default. Should be optional
4. if you think this would solve your case, I would be ok with:
a. adding an option to :or to override the default transformation logic with a custom one#2023-02-0612:00hanDerPederThanks for the thorough explanation and outlining the steps going forward.
I've gone through our usage of or-schemas. Knowing what I now know about the semantics, I've yet to find a single case where a multi spec isn't preferable. If I find one I'll send a PR.#2023-02-0610:57renewdoitI am trying to transform schemas, but due to the need for documentation, I have to read the source code; one potential problem is there is no API namespace to mark which function/protocol is considered internal and subject to change.#2023-02-0612:52ikitommiGood point. All protocols are part of the extender API. See https://github.com/metosin/malli#alpha, could add that to docs. Not planning on breaking those, will bump a minor if those change. Add a proper description what changed#2023-02-0612:53ikitommithe section also defines the public / extender API difference.#2023-02-0613:50Ben SlessPlaying a bit with persisting schemas in datascript, going by the AST I find its structure is not very regular. A value can be a schema reference or a concrete value. This makes writing a decent schema for when things are components, references and unique a headache.
How would you approach this?#2023-02-0614:45refset> This makes writing a decent schema
(I assume you mean DataScript schema here)
I don't have an answer for you...but I guess it may help to clarify: are you specifically hoping to use Datalog queries to perform analysis on the Malli schema(s)?#2023-02-0615:07Ben Slessyes, specifically, I'm trying to "unify" different schema registries, assuming they specify an ad-hoc key as communication channel, to check for rules and gain insights#2023-02-0615:08Ben Slessfor example, you send a message over a queue and say it's a string which starts with "abc". I read that message and say it's a string. We can unify these constraints#2023-02-0615:08Ben Slessbut to do that with complex schemas I need a way to store them reasonably in a datalog database#2023-02-0616:29refsetCool, that sounds pretty interesting! If you don't have a massive graph, I wonder whether you may have an easier time with using (e.g.) Meander directly on the Malli schemas ...although I've not actually used Meander before, I'm just speculating 🙂#2023-02-0616:42Ben SlessRecursive unification with meander is hard#2023-02-0616:43Ben SlessSpeaking from experience#2023-02-0617:01Ben SlessI do think this is a datalog shaped problem#2023-02-0618:23sun-oneFirst off amazing library, it's been a pleasure to use.
My question is related to massive schemas. How do you store and use them? Currently I'm generating malli schemas off of a standard called FHIR which has massive amounts of data describing the datas schemas here's a sample of the size https://gitlab.com/genfhi/genfhi/-/blob/malli_changes/cljc/generator/resources/structure_definitions/structure_definitions/hl7/profiles-resources.json 30+mb json file that defines schema for various data types.
I generated a massive clj file https://gitlab.com/genfhi/genfhi/-/blob/schema_alts/cljc/generator/src/gen_fhi/generator/r4/schemas.cljc but unsurprisingly it gets killed on AOT compilation with method size too large (note this works just fails during AOT compilation).
Should I store these massive schemas on edn file and read-string them in (related to that I have another issue that comes up in that I have functions that would need to be evaled so I need to read the edn than eval it). Is this the right approach to be generating a giant map of schemas (all these data types ref one another).#2023-02-0618:26sun-oneI could split these into chunks of data that get merged at the end too. But just curious to hear if this is the right approach for schemas of this size.#2023-02-0719:19ikitommithanks! I get "...structure_definitions/hl7/profiles-resources.json" did not exist on "master"#2023-02-0719:20ikitommiIf you could share the files, happy to take a look what can be done#2023-02-0819:42sun-oneOh sorry I moved the branch let me update those links#2023-02-0819:49sun-one@U055NJ5CC links are updated here's the json file representing the source of schemas and my generated malli schema map.
https://gitlab.com/genfhi/genfhi/-/blob/malli_changes/cljc/generator/resources/structure_definitions/structure_definitions/hl7/profiles-resources.json https://gitlab.com/genfhi/genfhi/-/blob/schema_alts/cljc/generator/src/gen_fhi/generator/r4/schemas.cljc
I suspect I should be reading it from edn and doing the neccessary wiring with sci (I do use dynamic bindings which I'm assuming wouldn't be an issue). Validation was working for me when I run it on dev it's really just the AOT compilation that was causing me issues (I could maybe hack around this and do a trick to avoid AOT comp for this massive map).#2023-02-0622:04escherizeDo time schemas have generators? I couldn’t find anything on that#2023-02-0623:36escherizeMy colleague found:
'[malli.experimental.time :as mtime]
'[malli.experimental.time.generator :as timegen]
#2023-02-0705:22Ben SlessIs it missing from the documentation?#2023-02-0716:24escherizeProbably not#2023-02-0716:25escherizeIn fact: https://github.com/metosin/malli#generators---malliexperimentaltimegenerator it’s right here#2023-02-0623:36escherizeMy colleague found:
'[malli.experimental.time :as mtime]
'[malli.experimental.time.generator :as timegen]
#2023-02-0705:59renewdoitm/entries wraps the entries into [::m/varl :int] like reified structure, how do I unwrap the entry get that :int out?#2023-02-0706:06renewdoitAnswer by my own: m/children can access it.#2023-02-0717:37marrsI'm trying to create a schema for a map that can contain one of a number of keys, e.g. a login form schema that can accept :username or :email. I'm sure it must be possible to create a schema to represent this but I can't find mention of it in the docs. I want to write something like
(def schema
[:map
[:or
[:username [:string]]
[:email [:string]]
[:password [:string]]])
#2023-02-0718:36Noah BogartCould use an :fn schema:
(def my-schema
[:and
[:map
[:x int?]
[:y int?]]
[:fn (fn [{:keys [x y]}] (or x y))]])#2023-02-0718:43marrsThanks, I'll think about that#2023-02-0718:44escherizemulti schema?#2023-02-0718:45escherizenah, that’s not quite right ^#2023-02-0718:46escherizeI’d go with @UEENNMX0T’s fn approach#2023-02-0719:15ikitommiYes, there is no declarative syntax for dependent keys. One could cook up that with :multi but would be quite ugly. This could be nice (and doable in the user space):
(def schema
[:and
[:map
[:username {:optional true} [:string]]
[:email {:optional true} [:string]]
[:password [:string]]]
[:keys/xor [:username :email]]])#2023-02-0719:16ikitommibut, not sure if that is worth it. :fn is quite clear already.#2023-02-0719:45escherizekind of rhymes with the https://github.com/bsless/malli-keys-relations library#2023-02-0720:35ikitommiyes, tried to find that as an example here, didn’t, thanks!#2023-02-0809:33marrsThanks, everyone. I ended up making 2 different schemas and choosing between them at runtime depending on which field was presented in the submitted form.#2023-02-0809:45marrs> Yes, there is no declarative syntax for dependent keys. One could cook up that with :multi but would be quite ugly. This could be nice (and doable in the user space):
> (def schema
> [:and
> [:map
> [:username {:optional true} [:string]]
> [:email {:optional true} [:string]]
> [:password [:string]]]
> [:keys/xor [:username :email]]])
>
The only thing about this (and its equivalent :fn version) is what would happen if neither :username nor :email were submitted. Both fields are declared as optional which implies that the form should validate, but that's not correct. Each field is required if the other isn't present.#2023-02-0723:16PanelHi, I hack something up to convert malli schema to EQL querry, it support recursive query.
Here's the code with an example at the bottom that convert the burger schema from http://malli.io
(ns malli-eql
(:require
[malli.core :as m]
[malli.util :as mu]))
(defn trim-branch-path [paths]
(reduce (fn [acc item]
(if (some #{item} (map #(vec (take (count item) %)) acc))
acc
(conj (vec (remove #(#{(butlast item)} %)
acc))
item)))
[]
paths))
(defn -collect [schema]
(let [state (atom {})]
(m/walk
schema
(fn [schema _ _ _]
(let [properties (m/properties schema)]
(doseq [[k v] (-> (m/-properties-and-options properties (m/options schema) identity) first :registry)]
(swap! state assoc-in [:registry k] v))
(swap! state assoc :schema schema)))
{::m/walk-schema-refs true})
@state))
(defn map-vec-tree-seq [form]
(tree-seq #(or (sequential? %)
(associative? %))
(fn [form]
(cond (sequential? form) (next form)
(associative? form) (#(interleave (keys %) (vals %)) form)))
form))
(defn recursive-registry [schema]
(->> (-> schema m/schema -collect :registry)
(map (fn [[k v]]
[k
(->> v
mu/subschemas
trim-branch-path
(map #(if (and (= (m/type (:schema %)) :ref)
(= k (first (m/children (:schema %)))))
(assoc % :in (conj (:in %) '...))
%))
(map :in))]))
(filter (fn [[_k v]] (some #{'...} (map-vec-tree-seq v))))
(into {})))
(defn path->eql [path]
(let [p' (->> path
(remove #{:malli.core/in}))
eql (reduce #(hash-map %2 %1) (if (= '... (last p')) (last p') [(last p')]) (reverse (drop-last p')))]
(or (if (vector? eql)
(first eql)
eql) [])))
(defn deep-merge-eql [forms]
(reduce (fn f'
([] nil)
([a b]
(let [s (when (and (map? b) (sequential? a))
(some
(fn [x'] (when (get x' (ffirst b))
x'))
a))
r (cond (and (vector? a)
(vector? b))
(vec (into #{} (concat a b)))
s (conj (vec (remove #{s} a)) (deep-merge-eql [s b]))
(and (vector? a)
(map? b)
(some #{(ffirst b)} a)) (conj (vec (remove #{(ffirst b)} a)) b)
(and (vector? a)
(map? b)) (conj a b)
(and (map? a)
(map? b)
(= (ffirst a) (ffirst b))) (merge-with f' a b)
(and (map? a)
(map? b)) [a b]
(and (vector? b)
(map? a)
(some #{(ffirst a)} b)) (conj (vec (remove #{(ffirst a)} b)) a)
(and (map? a)
(vector? b)) (conj b a)
(vector? a) (vec (into #{} (conj a b)))
(vector? b) (vec (into #{} (conj b a)))
:else
(vector a b))]
r)))
forms))
(defn schema->paths [schema registry]
(let [paths (atom [])
collect-paths-walker! (fn walker'
([s] (walker' s []))
([s parents]
(m/walk s (fn [schema path _children _options]
(let [is-ref (and (m/-ref-schema? schema)
(m/-ref schema))
ref-name (when is-ref (m/-ref schema))
in (mu/path->in (m/schema s) path)]
(cond (and is-ref
(some #{ref-name} (keys registry)))
(mapv #(swap! paths
conj
(vec (concat parents in %)))
(get registry ref-name))
(and is-ref ref-name)
(walker' (m/deref schema) (vec (concat parents (mu/path->in (m/schema s) path))))
:else (swap! paths conj (vec (concat parents in)))))))))]
(collect-paths-walker! schema)
@paths))
(defn schema->eql [schema]
(->> (schema->paths schema (recursive-registry schema))
trim-branch-path
(map path->eql)
deep-merge-eql))
(schema->eql [:schema
{:registry {"Country" [:map
{:closed true}
[:name [:enum :FI :PO]]
[:neighbors
{:optional true}
[:vector [:ref "Country"]]]],
"Burger" [:map
[:name string?]
[:description {:optional true} string?]
[:origin [:maybe "Country"]]
[:price pos-int?]],
"OrderLine" [:map
{:closed true}
[:burger "Burger"]
[:amount int?]],
"Order" [:map
{:closed true}
[:lines [:vector "OrderLine"]]
[:delivery
[:map
{:closed true}
[:delivered boolean?]
[:address
[:map
[:street string?]
[:zip int?]
[:country "Country"]]]]]]}}
"Order"])
;; => [{:lines
;; [:amount
;; {:burger [:description :name {:origin [:name {:neighbors ...}]} :price]}]}
;; {:delivery
;; [:delivered {:address [:street :zip {:country [:name {:neighbors ...}]}]}]}]#2023-02-0923:32JoelWow this is eery, I came here to ask a question about building some pathom code from malli. Are you working on something open-source and using pathom?#2023-02-0923:38PanelI’m also playing with pathom and malli, nothing worth publishing atm.
But I think it might be useful to have more tool to transform malli schema into other form to be used to generate form/pathom/db schema/…
The actual part relevant to eql in the above code is not much.#2023-02-0923:40Joeli had written code to pull data from a json request using malli, which i realized is too naiive (like nested data). But, also wanting to pass that to pathom to query as well.#2023-02-0923:42Joeli’m putting a “pathom like attribute” in the properties sections in malli schema to do the mapping, but the ::pco/output for pathom needs a more accurate mapping.#2023-02-0923:43Joelis that what you are generating above?#2023-02-0923:49PanelThe code is generating the query, but it could be adapted to generate pathom output form easily. But those would be dependent on the underlying data store, so for example the above burger example could be split into many different resolver.#2023-02-1000:36Michael Gardneris there an established "best practice" re: calling m/schema or not in your schema defs? E.g.
(def foo-schema
[:map ...])
vs
(def foo-schema
(m/schema
[:map ...]))
So far I've been writing the latter to get fail-fast behavior when a schema is invalid, but it's more verbose and forces the use of malli.util fns to transform/combine schemas. And AFAIK there's only a small performance benefit to calling m/schema thanks to Malli's caching. So I'm not sure what to pick as a default.#2023-02-1001:42rutledgepaulvOne suggestion could be to write a macro which defs and validates a schema but defines the value of the var as the original data as long as validation succeeds#2023-02-1001:43rutledgepaulvOr a function that does the same and just use that instead of m/schema in your second example#2023-02-1018:22Michael GardnerI could certainly do that, although it's still a little more verbose. And I'm not even sure I need it given that my data schemas will end up getting validated at ns load time via the function schemas that use them anyway.
So I guess my only remaining question is about performance. Is it true that there's no meaningful performance benefit to calling m/schema any more?#2023-02-1114:40Varghiz cljHi,
We've a use case where we need to apply schema validation on the body based on the type value in the header.
For example if the type is foo then should apply schema/foo . If the type is bar then should apply schema/bar.. dynamically#2023-02-1114:49Ben SlessYou can always add a middleware which adds it to the body from header before coercion#2023-02-1114:55Varghiz cljThank you @UK0810AQ2..I'm new to malli & reitit, can you please give me a reference or sample on middleware#2023-02-1114:55Ben SlessYou're using malli in http, are you using it with reitit?#2023-02-1114:57Varghiz cljI use malli for schema validation and reitit.swagger for API definition and routes.#2023-02-1115:06Ben SlessHow does your ring handler look like?#2023-02-1115:15Varghiz clj["/dosomething"
{:swagger {:tags ["do something"]}
:post {:summary "route the call after validating the payload based on foo or bar"
:scope [:api:write]
:parameters {:header [:map
[:authorization schema/authorization]
[:type schema/type-header]] ;; foo or bar
:body [:map
[:events schema/general-body]]} ;; this is the part we need to call schema/foo or schema/bar based on type
:handler (h/process-api config)}}]#2023-02-1115:15Varghiz cljthe part events is a generic such that based on the type, we will apply one schema vs the other#2023-02-1115:18Ben SlessThat's not the ring handler, just reitit routes#2023-02-1115:19Varghiz cljoops.. sorry#2023-02-1115:24Varghiz clj(defn process-api [config]
(fn [{{:keys [header body]} :parameters}]
(let [events (body :events)
total (count events)]
(let [results (k/send-events config events header)
successful (count (remove :error results))]
{:status (cond
(zero? successful) 400
(= total successful) 200
:else 206)
:body {:events results}}))))#2023-02-1116:20Ben SlessYou should share your entire code, what about reitit.ring/ring-handler#2023-02-1221:41steveb8nQ: should a uuid decoded by transit satisfy the :uuid built-in schema? I found it does not on cljs. Bug or feature?#2023-02-1315:12SigveHi, is it possible to use a schema with unique tagged values to retrieve the the corresponding field in a value that validates against that shcema?
e.g.
(def Nested [:map
[:a
[:map [:b
[:map [:c
[:map [:d :string]]]]]]]])
(m/walk-and-get Nested :d {:a {:b {:c {:d "the value of d"}}}}) ;=> should return "the value of d"#2023-02-1315:18Braden ShepherdsonI'm getting errors while loading malli.generate, with advanced CLJS compilation via shadow-cljs, running tests in Node.
the code in question (with --pseudo-names so they don't get shortened) is:
$CLJS.$malli$core$t_malli$0core69071$$.prototype.$malli$core$RefSchema$_ref$arity$1$ = $CLJS.$JSCompiler_unstubMethod$$(75, function() {
return null;
});
failing because $CLJS.$malli$core$t_malli$0core69071$$ is undefined, so there's no prototype to access. I'm still debugging, but does anyone have any insight? is this supposed to work?#2023-02-1315:23Braden Shepherdsonit looks like it's a problem with reify under CLJS advanced compilation; see this open bug https://clojure.atlassian.net/browse/CLJS-3207#2023-02-1315:29Braden Shepherdsonhere's part of the the generated code. it's pretty gnarly, but the upshot is that reify is conditionally creating new classes only once, to make sure they don't get redefined. but that means the (would-be global) prototypes don't exist until the function containing the reify is called at least once.
$malli$core$_val_schema$cljs$0core$0IFn$0_invoke$0arity$00$$ = function() {
if ("undefined" === typeof $CLJS.$malli$$ || "undefined" === typeof $malli$core$$ || "undefined" === typeof $CLJS.$malli$core$t_malli$0core69063$$) {
$CLJS.$malli$core$t_malli$0core69063$$ = function($meta69064$$) {
this.$meta69064$ = $meta69064$$;
this.$cljs$lang$protocol_mask$partition0$$ = 393216;
this.$cljs$lang$protocol_mask$partition1$$ = 0;
}, $CLJS.$malli$core$t_malli$0core69063$$.prototype.$cljs$core$IWithMeta$_with_meta$arity$2$ = function($_69065$$, $meta69064__$1$$) {
return new $CLJS.$malli$core$t_malli$0core69063$$($meta69064__$1$$);
}, $CLJS.$malli$core$t_malli$0core69063$$.prototype.$cljs$core$IMeta$_meta$arity$1$ = function() {
return this.$meta69064$;
}, $CLJS.$malli$core$t_malli$0core69063$$.prototype.$malli$core$AST$$ = $CLJS.$cljs$core$PROTOCOL_SENTINEL$$, $CLJS.$malli$core$t_malli$0core69063$$.prototype.$malli$core$IntoSchema$$ = $CLJS.$cljs$core$PROTOCOL_SENTINEL$$, $CLJS.$malli$core$t_malli$0core69063$$.prototype.$malli$core$IntoSchema$_type$arity$1$ = function() {
return $CLJS.$cljs$cst$keyword$malli_DOT_core_SLASH_val$$;
}, $CLJS.$malli$core$t_malli$0core69063$$.prototype.$malli$core$IntoSchema$_type_properties$arity$1$ = $CLJS.$JSCompiler_stubMethod$$(25), $CLJS.$malli$core$t_malli$0core69063$$.prototype.$malli$core$IntoSchema$_into_schema$arity$4$ = function($form$jscomp$16_parent$jscomp$56$$, $properties$jscomp$27$$, $children$jscomp$33_schema__$1$jscomp$32$$, $options$jscomp$124$$) {
var $parent__$1$jscomp$9$$ = this, $children__$1$jscomp$7$$ = $CLJS.$malli$core$_vmap$cljs$0core$0IFn$0_invoke$0arity$02$$(function($p1__69053_SHARP_$$) {
return $CLJS.$malli$core$schema$$.$cljs$core$IFn$_invoke$arity$2$ ? $CLJS.$malli$core$schema$$.$cljs$core$IFn$_invoke$arity$2$($p1__69053_SHARP_$$, $options$jscomp$124$$) : $CLJS.$malli$core$schema$$.call(null, $p1__69053_SHARP_$$, $options$jscomp$124$$);
}, $children$jscomp$33_schema__$1$jscomp$32$$);
$form$jscomp$16_parent$jscomp$56$$ = new $CLJS.$cljs$core$Delay$$(function() {
return $malli$core$_simple_form$$($parent__$1$jscomp$9$$, $properties$jscomp$27$$, $children__$1$jscomp$7$$, $CLJS.$malli$core$_form$$, $options$jscomp$124$$);
});
$children$jscomp$33_schema__$1$jscomp$32$$ = $CLJS.$cljs$core$first$$($children__$1$jscomp$7$$);
var $cache$jscomp$18$$ = $malli$core$_create_cache$$();
if ("undefined" === typeof $CLJS.$malli$$ || "undefined" === typeof $malli$core$$ || "undefined" === typeof $CLJS.$malli$core$t_malli$0core69071$$) {
$CLJS.$malli$core$t_malli$0core69071$$ = function($meta69064$jscomp$1$$, $parent$jscomp$57$$, $properties$jscomp$28$$, $children$jscomp$34$$, $options$jscomp$125$$, $form$jscomp$17$$, $schema$jscomp$2$$, $cache$jscomp$19$$, $meta69072$$) {
this.$meta69064$ = $meta69064$jscomp$1$$;
this.parent = $parent$jscomp$57$$;
this.$properties$ = $properties$jscomp$28$$;
this.children = $children$jscomp$34$$;
this.options = $options$jscomp$125$$;
#2023-02-1315:46Braden ShepherdsonI'm experimenting with adding #(:cljs ...) code to malli.core to touch these reify'd classes eagerly, since that apparently prevents the issue from happening; see the linked bug.#2023-02-1413:52Bingen Galartza IparragirreHello!
I'm having some trouble to define a custom registry in Malli
(def custom-registry
{::decimal [:double {:min 2}]})
(def registry
(-> custom-registry
(mr/registry)
(mr/composite-registry m/default-registry)))
(m/validate [::decimal] 2.5 {:registry registry})
Running the above code throws the following exception: :malli.core/invalid-schema {:schema [:double {:min 2}]}
What am I doing wrong? Thx in advance#2023-02-1414:09dvingoIt's a bit confusing but storing vector schemas in the registry and using vector lookup syntax ins't valid. I had the same thought and opened a PR to "fix" it
https://github.com/metosin/malli/pull/848
but it's not a bug, it's how schemas work. You can get around it using either [:ref ::decimal] or [:schema ::decimal]
Maybe the docs should be updated to have a note about it#2023-02-1414:09delaguardoyou registry is fine, the problem is in [::decimal] . This is invalid syntax for malli.#2023-02-1414:09delaguardo(m/validate ::decimal 2.5 {:registry registry}) works just fine#2023-02-1414:16dvingoit's confusing because [:int] is valid, but refs in vectors are not#2023-02-1414:18Bingen Galartza IparragirreYou are right, just tested that (m/validate ::decimal 2.5 {:registry registry})` works. But what happens if I want to have a schema with children?
This fails again:
(def custom-registry
{::decimal [:double {:min 2}]
::object [:map]})
(def registry
(-> custom-registry
(mr/registry)
(mr/composite-registry m/default-registry)))
(m/validate [::object [:item-1 ::decimal]] {:item-1 2.5} {:registry registry})#2023-02-1414:20Bingen Galartza Iparragirre@U051V5LLP if I understood right the workaround you suggest doesn't work in this case, does it?#2023-02-1414:27dvingoyea, looks like it won't work in that case, I'm not sure what's the reasoning behind why though#2023-02-1414:35delaguardo> But what happens if I want to have a schema with children?
@UK14W219B you have to create a schema that can accept parameters. Adding just [:map] into the registry doesn't allow your ref schema to have children.
There is a way to add custom schema type - https://github.com/metosin/malli#custom-schema-types#2023-02-1414:37delaguardo@U051V5LLP [:int] is valid because :int schema can accept parameters like :min/:max#2023-02-1415:44Bingen Galartza IparragirreUnderstood! Thank you both a lot#2023-02-1416:24dvingo@U04V4KLKC it would be awesome, if you don't mind, to put together a little cheat-shee/table of what constitutes an invalid schema and when you have to create your own custom schema type vs just creating a ref schema. I think this would be a great addition to the readme :)#2023-02-1414:27dvingoI ported the new experemental.time schemas to cljs https://github.com/metosin/malli/pull/853#2023-02-1418:39Ben SlessThis is wonderful!
I'm glad it took only a small amount of changes#2023-02-1419:10dvingoyea! great work on the jvm version! Felt pretty magical changing a few clj files to cljc and a few imports and running the generators the first time in a cljs repl, ha#2023-02-1515:58Luke JohnsonGood day! Is there a way to pretty print a Malli schema? pprint is just one long line.#2023-02-1516:01delaguardomalli schema is data so any library that offers pretty printing for clojure data would do. For example https://github.com/kkinnear/zprint#2023-02-1516:08Luke JohnsonI suppose I was asking if there was a pre-built function ready for my use. pprint works fine for some schemas in my REPL, but I think the property maps and reader tags cause problems. And I was hoping not to have to roll my own solution, but that may be where I’m headed. Thanks.#2023-02-1516:10delaguardoCould you share an example when pprint doesn't work as you expected?#2023-02-1516:21Luke JohnsonComing up with an example, it seems that mu/optional-keys is causing problems.#2023-02-1516:22Luke JohnsonReally big schemas also break pprint, I think.#2023-02-1516:50delaguardouser=> (clojure.pprint/pprint (m/-form (mu/optional-keys (into [:map] (map (fn [ch] [(keyword (str (char ch))) :int]) (range (int \a) (int \z)))))))
[:map
[:a {:optional true} :int]
[:b {:optional true} :int]
[:c {:optional true} :int]
[:d {:optional true} :int]
[:e {:optional true} :int]
[:f {:optional true} :int]
[:g {:optional true} :int]
[:h {:optional true} :int]
[:i {:optional true} :int]
[:j {:optional true} :int]
[:k {:optional true} :int]
[:l {:optional true} :int]
[:m {:optional true} :int]
[:n {:optional true} :int]
[:o {:optional true} :int]
[:p {:optional true} :int]
[:q {:optional true} :int]
[:r {:optional true} :int]
[:s {:optional true} :int]
[:t {:optional true} :int]
[:u {:optional true} :int]
[:v {:optional true} :int]
[:w {:optional true} :int]
[:x {:optional true} :int]
[:y {:optional true} :int]]
nil
mu/optional-keys returns custom schema type. There is m/-form function that turn the type to its data representation. This data is suitable for pprinting#2023-02-1517:12Luke JohnsonThat’s really helpful! Thanks @U04V4KLKC#2023-02-1517:52Ben SlessDid you try calling malli.core/form on it first?#2023-02-1517:53Luke JohnsonTIL: m/form is what I needed!#2023-02-1517:53Luke JohnsonThanks!#2023-02-1520:09joshkhi’ve read through the docs and i’m sure that i missed the obvious 😇. how can i validate that a map’s keys are all keywords, and their values are all strings, given that the k/vs are unknown?#2023-02-1520:11joshkhand like magic, i found it after asking. :map-of#2023-02-1521:10Noah Bogarti love/hate when that happens. glad you found it!#2023-02-1521:11joshkhstory of my life 😎#2023-02-1713:25Bingen Galartza IparragirreHello! Any hints about how I could add a custom decoder to all the schemas of a certain type?
We have a custom registry defined like this:
{:object (m/-map-schema)}
so then we use :object in our Malli schemas.
Is it possible to add a custom decoder to all the :object schemas? without having to explicitly set it in all the schemas that use :object#2023-02-1713:58Bingen Galartza IparragirreI see the following could be a possible workaround:
(def test-schema
(m/schema
[::object
[:test
[::double]]]
{:registry {::object (m/-map-schema)
::double (m/-double-schema)}}))
(m/decode test-schema {:test 1.0}
(mt/transformer
{:name :test
:decoders {:map {:enter #(prn %)}}}))
But I would like to tie the tranformer to object and not map#2023-02-2006:25ikitommiHi, there is no easy to way to do this atm as the ::object is “just a map” after schema lookup . I could add support for :type-properties into m/-map-schema so this would work:
(def test-schema
(m/schema
[::object
[:test
[::double]]]
{:registry {::object (m/-map-schema {:type-properties {:decode/test {:enter #(prn %)}}})
::double (m/-double-schema)}}))
(m/decode
test-schema
{:test 1.0}
(mt/transformer {:name :test}))#2023-02-2006:25ikitommiif that would be ok, please write issue. fix = 1 line + test.#2023-02-2006:28ikitommiexplained: each IntoSchema can have (shared and immutable) type-properties - like Schema instances have properties. Schema application that need properties, can fallback to reading the properties from it’s parent.#2023-02-2013:02Bingen Galartza IparragirreYep, that would perfectly fit our use case. We will open an issue. Thank you#2023-02-2014:04Bingen Galartza IparragirreIssue opened: https://github.com/metosin/malli/issues/855#2023-02-2014:04Bingen Galartza IparragirreLet us know if you think we could help with a PR#2023-02-1714:39camdezHi all!
Having a bit of trouble with function instrumentation via metadata, maybe I’m missing something…
I’m using ! , and instrumentation works great initially, but is lost if I re-evaluate the function. My understanding was that the dev tooling would watch the var and re-instrument.
Is this normal? If so, any good solutions?#2023-02-1714:52steveb8nI had a similar experience. I found that rerunning dev/start refreshes it when registry and source change a lot in dev#2023-02-1714:54steveb8nIn my case my source is capturing the latest state of the registry and was not updating. I suspect the watcher assumes a particular style of registry and mine is not that#2023-02-1714:56steveb8nRerunning dev/start is a workaround that has worked for me. I added it to my repl fiddle forms i.e. everywhere I invoke the instrumented code#2023-02-1716:06camdezThanks. I similarly see that it picks up the changes on re-runs, but I’m aiming to just validation all function calls at development time and I was hoping to not have to litter my code with additional calls. Also I typically just re-evaluate individual functions.
Perhaps it creates a new var so it’s not instrumented?
I’m sure I can hook into CIDER or have a file-watcher based solution, I’m just not certain if that’s typically needed.
I appreciate the response!#2023-02-1717:47camdezAh, I see now Malli just watches the registry, not the vars in the registry. So it seems like things are working as expected, but it’s not convenient… so I either need a workaround or to switch to a different approach that touches the registry on re-eval.#2023-02-1717:58camdezIn a similar vein, if you want to instrument automated tests… any code with :malli/schema metadata not loaded at the time you enabled instrumentation would never get instrumented. So unless I’m missing something, it seems like the other instrumentation approaches are the way to go.#2023-02-1718:08camdezI suppose you could turn on the instrumentation in fixtures and it would work out fine.#2023-02-1717:34Thomas MoermanHi gang, a question regarding opinion:
• I implemented a custom malli schema using the IntoSchema and other protocols, inspired by the implementation of the standard library schemas (e.g :map, :map-of, etc.) I basically wanted a kind-of higher-order schema that (in function of its children/parameters) creates an internal schema, which is then used for further validation.
• My question: is implementing custom schemas something you would recommend against, or is this considered generally OK? E.g. from the point of view of Protocol stability of the current alpha version vs. a later version?
• If there's anyone who implemented their custom generic malli types, would you consider sharing the case/motivation?
Cheers!#2023-02-2006:18ikitommiMorning. It’s OK. Trying to keep protocols stable, haven’t changed much and if there is a good reason to change, it’s a minor bump (described in the CHANGELOG & https://github.com/metosin/malli#alpha documentation.#2023-02-2006:18ikitommie.g. part of the extender API#2023-02-2007:37Thomas MoermanGreat! Thanks for your feedback.#2023-02-1809:58pavlosmelissinosI want to define a schema for a map that can contain anything (it's a placeholder)
(every? true? [(m/validate schema {:foo "bar" "foo" :bar})
(m/validate schema {{:a :b :c :d} [:ho :ho :ho]})
(m/validate schema {})
(not (m/validate schema '()))
(not (m/validate schema #{}))
(not (m/validate schema []))])
;;=> true
[:map-of any? any?] seems to work; is it the right schema to use?#2023-02-1810:25Ben SlessWhy not just :map?#2023-02-1810:28pavlosmelissinosBecause I was sure I tried it and it didn't work facepalm#2023-02-1810:29pavlosmelissinosThanks, I suspected I was missing something obvious 🙂#2023-02-1810:54Ben SlessHappens 🙃#2023-02-2001:53JoelDoes walk not give the custom map data? Do I need to use m/ast for that? I thought “options” would represent the :label info i embedded.
(m/walk [:map
[:ts {:label :domain/ts} [:and {:label :domain/specific} :int [:> 0]]]]
(fn [s p c o] (println o) s))
#2023-02-2006:15ikitommiMorning. what do you mean with “custom map data”? the last argument is top-level options. if you want to read the schema properties, you should use (m/properties s). For map-entries, there are 2 options:
1. if the schemas has entries, e.g. m/entries returns non-nil, walk those too
2. use option {::m/walk-entry-vals true}
(m/walk
[:map
[:ts {:label :domain/ts} [:and {:label :domain/specific} :int [:> 0]]]]
(fn [s p c o] (some-> s m/properties println) s)
{::m/walk-entry-vals true})
;{:label :domain/specific}
;{:label :domain/ts}
;=> [:map [:ts {:label :domain/ts} [:and {:label :domain/specific} :int [:> 0]]]] #2023-02-2013:11hifumi123I’ve just checked bundle size for my cljs app I’ve noticed malli.core is the 2nd largest thing after clojurescript itself. Are there any tips to reducing the impact of malli on bundle size?#2023-02-2013:39ikitommireally good question! Malli is build for DCE, but multimethods can’t be DCEd. There are examples in malli repo (and a guide in README) for a minimal bundle, with just selected schemas and schema applications (validation). With the minimal set, Malli starts with 1.6KB zipped#2023-02-2013:41ikitommito disable default registry, you can:
:closure-defines {malli.registry/type "custom"}
… but then you have to pull the selected schemas at runtime:
(require '[malli.registry :as mr])
(mr/set-default-registry!
{:string (m/-string-schema)
:maybe (m/-maybe-schema)
:map (m/-map-schema)})#2023-02-2013:44ikitommiif you depend on an optional ns, like malli.generator - multimethod is mounted for ALL schemas, regardless if that’s needed or not. Not sure if there is a way to make the extensions slimmer.#2023-02-2013:44ikitommibut, code size is a priority issue. all help & issues welcome.#2023-02-2013:46ikitommiIf all schemas had a concrete class impl, e.g. MapSchema, AndSchema etc, the DCE might work also for the extension cases. Not sure. If it did, would be open to changing this in the core.#2023-02-2013:48ikitommihttps://github.com/metosin/malli/blob/master/app/malli/app2.cljc#2023-02-2015:31hifumi123Super helpful information! This allowed me to reduce malli's footprint from 139.75 kb to 75.09 kb 🙂 #2023-02-2015:32hifumi123Thank you!#2023-02-2018:41jmmkI’m using malli to parse/validate a string format that has different types of separators
e.g. a/b/c/d e f g where part 1 has elements separated by / and each part is separated by " "
I see instaparse has the ability to ignore certain characters
> In instaparse, you can use angle brackets <> to hide parsed elements, suppressing them from the tree output.
Does malli have any such capability to ignore certain elements in the parsed output?#2023-02-2018:42jmmkIt’s not hard to “just ignore those keys” when pulling data out, but it’s a bit annoying to have to invent tags for useless data#2023-02-2018:43jmmke.g.
[:catn
[:element1 some-schema]
[:sep1 slash]
[:element2 some-schema]
[:sep2 slash]]
#2023-02-2018:46Ben SlessCall them :_#2023-02-2018:57jmmkmalli doesn’t allow repeated tags#2023-02-2018:57jmmkDuplicate key :_#2023-02-2019:20Ben Slessdamn, but at least you don't have to think about names then, :_0, etc#2023-02-2117:29eggsyntaxHi! Just started using malli in a project for the first time & am really happy with it so far 😍
One question -- is there an equivalent of spec's with-gen, so that I can validate and generate in different ways? In particular I have an :id spec that just needs to be a uuid to be valid, but which I'd like to generate as one of several known real ids.#2023-02-2117:31Ben Slessgen/gen property in schema#2023-02-2117:31eggsyntaxAwesome, thanks!#2023-02-2117:46eggsyntaxBingo! That was easy 🙂
Whew, it's been a while since I've thought about test.check.generators, though, fun to dig back into that...#2023-02-2118:07ikitommi:gen/schema is also nice, all generator-related properties are described here: https://github.com/metosin/malli#value-generation#2023-02-2121:05escherize:gen/elements is also useful when there’s a constrainted or annoying-to-generate shape.#2023-02-2121:10eggsyntaxOh, :gen/elements is handy, I hadn't caught that; I'd instead done
(def UserType
[:map
[:id {:gen/gen [:enum fixed-seeds/user-type-ids]}]
...])
Thanks, y'all!#2023-02-2120:27David GI'm trying to do a comparison for my team, the pros and cons for spec vs malli.
Did anyone make a comprehensive comparison? I see https://github.com/kennyjwilli/specalli as something that can be used to automatically compare the syntax, there's also plenty of benchmarks available.
Does anyone know of any github pages that go into other comparison details.#2023-02-2214:44Noah BogartY'all know about https://github.com/borkdude/speculative? It was an attempt at adding clojure.spec specs for nearly all of the clojure.core functions, both as a means of documentation and a way to check input-output when doing development. It stalled out as clojure.spec is not performant enough. now that I've added instrumenting vars in other namespaces, I wonder if we could do a malli version of speculative, see if the performance is any better/if it would help uncover spaces where malli could improve#2023-02-2214:47borkdudespeculative still has its use, e.g. in https://borkdude.github.io/re-find.web/
I think even with a malli speculative, instrumenting core would just be way too slow and probably not worth it#2023-02-2214:47Noah Bogartoh cool! I didn't realize you were using it for re-find, I thought you were using something like grasp.#2023-02-2214:48borkdudea good test case would be advent of code puzzles. at one point I had a solution which took 15 minutes with instrumentation and 50ms without or so#2023-02-2214:49borkdude(exact number I don't remember but it was something absurd like that)#2023-02-2214:49Noah Bogartyeah, it's debatable if it would be worth it, but malli is pretty dang fast comparatively: 40ns vs 450ns for simple schemas. 10x speed increase is sharp#2023-02-2214:49borkdude10x won't really solve that problem imo#2023-02-2214:50borkdudethe problem is that some core functions will become 1000x slower#2023-02-2214:50borkdudebut exact benchmarks must be done to revalidate that, it's been a while#2023-02-2219:33JoelHow do validate that I have a valid malli (vector) schema? Or, what is the schema of schemas?
eg.
[:map
[:attribute :keyword]
[:schema [:and :vector :schema]]
.core/child-error {:type :schema, :properties nil, :children nil, :min 1, :max 1}#2023-02-2219:58frankthere's a malli.core/schema? that looks promising#2023-02-2220:12JoelLooks like that checks to see if it’s an instance of a schema, which would force me to invoke (schema) on the data. I’ve been on the fence on how proactive I should be about doing that. I do see there is (form) which does get the data back… I was expecting that there was a schema schema somewhere though.#2023-02-2220:13frankah I see#2023-02-2301:44dvingoI've just wrapped (malli.core/schema <args>) in a try catch - catch branch returns false to determine if a valid schema is passed in#2023-02-2319:13steveb8nQ: I’m using instrumented fns and they are logging invalid data correctly but I’m drowning in logs from this in some cases. Is there a way to make an invalid fn input throw an exception i.e. halt on first bad value detected instead of continuing?#2023-02-2319:16steveb8nHmm, I see the default report behaviour is m/-fail so it must be my code. I’m using core.async with the go-try macro from org.clojars.akiel/async-error#2023-02-2319:17steveb8nmaybe it’s go-try that is continuing processing when Malli has thrown?#2023-02-2319:24escherizeAre you using mx/defn?#2023-02-2319:24steveb8nnope: using m/=>#2023-02-2319:25steveb8nbut I’m not using a global/default registry so that’s a bit different to the way the docs show it#2023-02-2319:25steveb8nI have a custom registry which I provide to the m/=> form#2023-02-2319:26escherizeAnd you need it to just throw when an error occurs instead of log?#2023-02-2319:26steveb8n(btw hi Brian. it’s been a while 🙂#2023-02-2319:26steveb8nyes, exactly#2023-02-2319:26escherizeHey Steve 🙂#2023-02-2319:26escherizeOk, then you can instrument it with a custom report fn. https://github.com/metabase/metabase/blob/master/src/metabase/util/malli.cljc#L93-L95#2023-02-2319:26steveb8noh cool. I’ll try that#2023-02-2319:26escherizeheres the one we are using. explain-fn-fail! throws#2023-02-2319:27escherizebut any reporter that throws should solve it#2023-02-2319:27escherizeIt’s been a while!#2023-02-2319:38steveb8noh, I found my mistake. I was using pretty/reporter when I should have been using pretty/thrower#2023-02-2319:38steveb8nthanks anyway for the nudge. that helped me find the best option#2023-02-2321:08Brett RowberryNo rush. I was wondering if there’s a release planned that includes the fixes for https://github.com/metosin/malli/issues/819#2023-02-2609:49ikitommiWill cut a release next week#2023-02-2323:23Jakub ŠťastnýHey guys. I'm looking at the screenshot in the README (https://raw.githubusercontent.com/metosin/malli/master/docs/img/malli-defn.png) and it's precisely what I'm trying to achieve.
I have this:
(m/=> qid-str [:=> [:cat :int] :int])
(defn qid-str [version-number-list] (str/join "." version-number-list))
(qid-str [1 2 3 4])
But it's not doing anything. How do I hook up the schema validation into the fn execution, just like it's in the screenshot?#2023-02-2404:00escherizedid you call (dev/start)?#2023-02-2404:00escherizeYou could also look into using mx/defn#2023-02-2404:01escherizealso I think it’s:
(m/=> qid-str [:=> [:cat [:sequential :int]] :int])#2023-02-2414:15Jakub ŠťastnýWhere does (dev/start) come from?#2023-02-2414:17Jakub ŠťastnýThat sounds like some HTTP thing. I just have a very simple CLI app.#2023-02-2414:18camdezHey Jakub. I just went through this earlier in the week and it was rather confusing for me (but I got there!).
Here’s what I (and perhaps you) need(ed) to know:
1. malli.instrument/collect! needs to be called to find all of the function annotations in the namespaces you tell it, or the current namespace by default.
2. malli.instrument/instrument! then rewrites the functions to include the validation you’re looking for.
3. ! will do both of these things for you.
4. It also installs a watcher that will do that rewriting process automatically when m/=> is evaluated.
5. Contrary to the docs, the order of the function and the m/=> matters (a lot) for this case. If you eval the m/=> first, and then the function (as will happen in your code), the function will be redefined without the validation.#2023-02-2414:20Jakub Šťastný@U0CV48L87 thanks Cameron, that's very helpful.#2023-02-2414:21camdezHappy to help. Let me know if that gets it working for you.#2023-02-2414:24camdezOne more gotcha to watch out for: if namespaces aren’t loaded (yet) then they’re not going to get instrumented. It’s less of a problem when using , since you can work around it using m/=>, it can be a headache in tests or if you’re trying to use the function metadata annotations.
I haven’t gotten to a setup I’m entirely happy with for both interactive development and testing.#2023-02-2414:25camdezUsing the macro annotation form should solve all of the issues, but I was hesitate to adopt it first.#2023-02-2414:26Jakub ŠťastnýGood to know. It's a lot to digest (as I'm generally fairly new to Clojure, though not to programming). Malli does look great.#2023-02-2414:51Jakub ŠťastnýOh wow good note on the namespaces!#2023-02-2418:00escherize> Where does (dev/start) come from?
It is the requires in the screenshot you posted.#2023-02-2416:33eggsyntax@ikitommi FYI big new Bollywood blockbuster 😁
https://www.youtube.com/watch?v=llkjMNOYQA8#2023-02-2722:51nivekuilis there a builtin to normalize map schema, like [:map :user/id [:foo :int]] => [:map [:user/id :int] [:foo :int]]#2023-02-2810:51Thomas MoermanQ: is there a recommended way to specify in Malli that something is to be a json-encoded string? Motivation: i want to use a json-encoded string as a GET query param using Reitit. It kinda works with a hacky workaround, specifying [:map [:filter [:maybe [:or [:map :str]]]]] as the schema, so that Swagger understands it as a map (but submits it as a string). Any other approach I might want to try? Cheers.#2023-02-2810:54Thomas Moermanwithout :or :string I'm getting a coercion error server side#2023-02-2811:29Ben SlessAdd a decoder for this parameter which parses it before it is validated#2023-02-2813:29Thomas MoermanThis appears to work, found the pointer in another 🧵. Thanks!
(m/-simple-schema
{:type :map
:pred map?
:type-properties
{:decode/string #(try
(-> %
(json/read-str)
(w/keywordize-keys))
(catch Exception _ %))}})
#2023-02-2813:35Ben SlessNo need to create a simple schema, just
[:map {:decode/string ,,,}]#2023-02-2813:36Ben SlessThen you can add any constraints you might have about the entries like you would a regular map, yes?#2023-02-2814:13Thomas MoermanGot it. 👍#2023-02-2814:20eggsyntaxI was a bit surprised by this behavior. I initially expected :* and :+ to produce similar results to :vector (although maybe as eg a list rather than vector), but it's very different.
user> (mg/generate [:* [:* [:* int?]]] {:seed 1, :size 3})
(0 -2 0 -1)
user> (mg/generate [:vector [:vector [:vector int?]]] {:seed 1, :size 3})
[[] [[0 -2 0] [-1]] []]
My guess is that what's going on is that :* and :+ just mean 'some number of instances of a type' without any implication of containment, and that the only reason we're seeing the outer list in the :* case is that if generate generates > 1 item it has to throw them in a sequence just so that it's a single return val.
Is that correct, or am I more fundamentally misunderstanding something?#2023-02-2815:02Ben SlessIt probably mimics the spec regex operators behavior which flatten by default#2023-02-2815:02Ben SlessIf you want to isolate them wrap them in a schema schema#2023-02-2815:03eggsyntaxYou know, I've used spec a bunch and I've somehow just never run across that flattening behavior. Oh well, always more corners to discover 😁#2023-02-2817:17elaroussHi, I am sorry if my question sounds stupid, I am not sure why mi/check is not working for me
I have this function with it’s schema:
(defn get-user! [id])
(m/=> get-user! [:=> [:cat uuid?]
[:map
[:user/first-name string?]
[:user/email [:and
{:gen/elements ["
First I run:
(dev/start! {:gen mg/generate :report (pretty/reporter)})
so then:
(get-user! (random-uuid)) ;; => #:user{:first-name "34Xdu9532PWK", :email "
But when I try:
(mi/check {:filters [(mi/-filter-var #{#'get-user!})]
:gen mg/generate
:report (pretty/reporter)})
mi/check does not generate a fake map value, but instead returns nil which causes the check to fail.
am I using this wrong? my goal is to use generative testing, but I couldn’t find a working example that uses the function schemas with something like defspec.#2023-03-0419:00ikitommioh, this is unfortunate, mi/check has a separate branch in mi/-strument so it ignores the :gen option. Please write an issue of this#2023-02-2822:57escherizeoverheard at work: is there a trick to get something like recursive seqexps with Malli?#2023-02-2822:58escherizee.g.
this schema works for generation, but not validation. is there a trick to get it working?
(let [sk [:schema {:registry {::filter
[:or
:boolean
[:cat [:= :and] [:* [:ref ::filter]]]]}}
::filter]]
(mg/generate sk)
;; => (:and false (:and (:and) (:and) true false))
(mc/validate sk true))#2023-03-0113:21ikitommiHi. recursive self-flattening regex schemas are not allowed. You can wrap the :ref with schema to make it work here:
(let [sk [:schema {:registry {::filter
[:or
:boolean
[:cat [:= :and] [:* [:schema [:ref ::filter]]]]]}}
::filter]]
(mg/generate sk)
;; => (:and false (:and (:and) (:and) true false))
(mc/validate sk true))#2023-03-0113:22ikitomminot sure if that’s clearly described in the README. Documentation improvements most welcome!#2023-03-0113:23ikitommiexample from readme:
(def Hiccup
[:schema {:registry {"hiccup" [:orn
[:node [:catn
[:name keyword?]
[:props [:? [:map-of keyword? any?]]]
[:children [:* [:schema [:ref "hiccup"]]]]]]
[:primitive [:orn
[:nil nil?]
[:boolean boolean?]
[:number number?]
[:text string?]]]]}}
"hiccup"])
(def parse-hiccup (m/parser Hiccup))
(parse-hiccup
[:div {:class [:foo :bar]}
[:p "Hello, world of data"]])
;[:node
; {:name :div
; :props {:class [:foo :bar]}
; :children [[:node
; {:name :p
; :props nil
; :children [[:primitive [:text "Hello, world of data"]]]}]]}]#2023-03-0120:05escherizeNice! Thanks. and Hi back 🙂#2023-03-0408:46pavlosmelissinosI've picked up from where https://github.com/metosin/malli/pull/211 left off because I'd love to have json-schema import for a project,
However, when trying to invert the tests in malli.json-schema-test, some of the tests fail because https://github.com/metosin/malli/pull/211/commits/fc5059b0886c20eb2497f3244b9a79eeec71d9d2malli converter recursively uses >`malli.util/update-properties` that apparently converts all the symbols to schema instances. Does anyone know why adding annotations is necessary at that point and if it's ok to do a single pass at the end instead?#2023-03-0408:50pavlosmelissinosan example to illustrate the issue:
(schema->malli {:type "string"})
;;=> string?
(= string? (schema->malli {:type "string"}))
;;=> false
(type (schema->malli {:type "string"}))
;;=> :malli.core/schema
(type string?)
;;=> clojure.core$string_QMARK___5475#2023-03-0408:59pavlosmelissinosOh, I see, I can't call it at the end because it has to get the annotations of each json-schema property.
Still, any suggestions for how to solve this? I suppose I could duplicate the expected pairs and modify the copy but it would be neat if I could reuse the existing one.#2023-03-0410:15pavlosmelissinosI've solved the false negatives by replacing
(is (= schema (sut/schema->malli json-schema))) with
(is (mu/equals (m/schema schema) (sut/schema->malli json-schema)))
but now the diff isn't as nice 😞
I'm no longer blocked duckie but I'd love to hear suggestions for improvements!#2023-03-0418:59ikitommiYes, you should check the forms, not instances. Some suggestion:
• would generate :string instead, I guess it contributes to better errors too
• You can just say (is (= :string (m/form (sut/...)))) to get proper diffs#2023-03-0421:13pavlosmelissinos> would generate :string instead, I guess it contributes to better errors too
>
Great feedback, thanks! I'm kinda new to malli so I'm still cargo-culting my way, sorry, 😅
> You can just say (is (= :string (m/form (sut/...)))) to get proper diffs
>
This works for strings but there isn't a :pos-int, right? So I'm "stuck" with pos-int? which has the same issue. Or do you mean it should generate something like [:int {:min 1}] or [:and :int [:> 0]] instead? Not sure what the idioms are.#2023-03-0516:58ikitommi🥳#2023-03-0523:31Piotr RoterskiHello 👋 I've just noticed that function schemas defined with metadata do not get properly reinstrumented by on ns reload. I've created this issue https://github.com/metosin/malli/issues/861 and put there a video showing this bug. Cheers!#2023-03-0700:52Patrick BrownI’m dealing with things I truly don’t understand. By virtue of how I was bootstrapping my system all my schemas are serializable and fit to be sent across the wire. Really, I just did that by doing what made sense. Wake up and Malli gift wrapped me a feature I’m unfamiliar with. I’m imagining the possibilities. Is anyone out there utilizing this feature, in what capacity? Sounds cool. Just curious.#2023-03-0715:05dgb23The more your program can be described as data (malli, reitit, integrant, deps.edn etc.) the easier it gets to manipulate your program through that data. You get easy opportunities to create all sorts of tooling.#2023-03-0719:08cap10morganI'm building a registry in one namespace (using a map of qualified keywords to vector-syntax schemas) that I would like to re-use schemas from a registry in another namespace. Is there a way to do that?#2023-03-0719:09JoelMore than just a simple merge?#2023-03-0719:11cap10morgan:thinking_face: I was hoping there was a way to reference only the ones I need, but I suppose that's just a call to select-keys away w/ merge. I'll give it a shot, thanks!#2023-03-0719:12cap10morganoh, I guess the other issue is the keys will still have the other namespace#2023-03-0719:12cap10morganbut that might be ok in this case...#2023-03-0719:13Joelalias ’em… eg. ::ns/in-schema1#2023-03-0719:13JoelAlso 1.11 let’s you have “fake” namespaces using as-alias fwiw.#2023-03-0719:13cap10morganyeah#2023-03-0719:14cap10morganI was hoping to bring them into this registry, namespace and all#2023-03-0719:14cap10morganbut it might be fine#2023-03-0719:17cap10morganactually, yeah, it is. I can just do ::this ::other/this to accomplish what I want w/ your merged registry suggestion. thanks!#2023-03-0820:08JoelI’m using mt/key-transformer {:encode keyword}, but I only want to decode keys to keywords if the corresponding schema key is a keyword (seems like this should be default)?#2023-03-0820:22JoelI suppose all-or-none could work in my case. So I could “query” the schema if it’s using keyword keys..
It looks like :encode is only handed the key so it wouldn’t have the context to decide what to turn it into (keyword or leave as-is).#2023-03-1007:53ikitommiby default, it encodes/decodes all keys. the transformers can see the schema, so it’s not hard to create a variant / option just to use the keys defined. m/strip-extra-keys-transformer is an example of looking into defined keys#2023-03-0916:41emccueRandom bit of info - you can use JFR to listen for class redefinitions. Might this be usable for detecting reloaded namespace vars?#2023-03-0916:41emccuepublic static void startJfr() throws IOException, ParseException {
var path = Paths.get("jfr.xml");
var config = Configuration.create(path);
var events = List.of("jdk.SocketRead", "jdk.SocketWrite", "jdk.TLSHandshake");
Runnable task = () -> {
try (EventStream es = new RecordingStream(config)) {
events.forEach(event -> es.onEvent(event, System.out::println));
es.start();
}
};
new Thread(task).start();
}#2023-03-0916:41emccue<configuration version="2.0">
<event name="jdk.SocketRead">
<setting name="enabled">true</setting>
<setting name="stackTrace">true</setting>
<setting name="threshold" control="socket-io-threshold">0 s</setting>
</event>
<event name="jdk.SocketWrite">
<setting name="enabled">true</setting>
<setting name="stackTrace">true</setting>
<setting name="threshold" control="socket-io-threshold">0 s</setting>
</event>
<event name="jdk.TLSHandshake">
<setting name="enabled">true</setting>
<setting name="stackTrace">true</setting>
<setting name="threshold" control="socket-io-threshold">0 s</setting>
</event>
</configuration>#2023-03-0916:41emccuejdk.ClassRedefinition
#2023-03-0916:41emccuespecifically this event^#2023-03-1007:48ikitommi👀#2023-03-0919:12dHi, there! Does malli have a function like spec2's select? Something like this maybe, supporting selecting nested keys:
(let [schema [:map
[::name string?]
[::age pos-int?]
[::address
[:map
[::address.city string?]
[::address.country string?]]]]]
(m/select schema [::name ::address {::address [::address.city]}]))#2023-03-0919:15Noah Bogarthttps://clojurians.slack.com/archives/CLDK6MFMK/p1658265412643769?thread_ts=1641570875.007400&channel=CLDK6MFMK&message_ts=1658265412.643769#2023-03-0919:17dThanks! I implemented my own function, looks very much like this one. Just wonder why this can't be on malli itself, as it seems some people(including me) think it is useful#2023-03-1007:47ikitommiThis would be nice, can’t recall the difference of EQL and Spec Select syntax, are they the same?#2023-03-1007:48ikitommianyway, here’s the issue https://github.com/metosin/malli/issues/724#2023-03-1007:57ikitommiFiguring out best way to https://github.com/metosin/malli/issues/264. While doing, about to break “content-dependent schema creation” using a new :compile prop instead of a top-level callback function. Look like this:
(def Between
(m/-simple-schema
{:type :user/between
:compile (fn [_properties [min max] _options]
(when-not (and (int? min) (int? max))
(m/-fail! ::invalid-children {:min min, :max max}))
{:pred #(and (int? %) (>= min % max))
:min 2 ;; at least 1 child
:max 2 ;; at most 1 child
:type-properties {:error/fn (fn [error _] (str "should be betweeb " min " and " max ", was " (:value error)))
:decode/string mt/-string->long
:json-schema {:type "integer"
:format "int64"
:minimum min
:maximum max}
:gen/gen (gen/large-integer* {:min (inc min), :max max})}})}))
(m/form [Between 10 20])
; => [:user/between 10 20]
(-> [Between 10 20]
(m/explain 8)
(me/humanize))
; => ["should be betweeb 10 and 20, was 8"]
(mg/sample [Between -10 10])
; => (-1 0 -2 -4 -4 0 -2 7 1 0)#2023-03-1007:58ikitommiMR here for comments https://github.com/metosin/malli/pull/866#2023-03-1008:22CaseyI have an annoying input to a value in my system. It is a vector of items. But if there is only a single item, I receive the one item, not wrapped in a vector. Is there a built in way with malli to handle this? I want the single item to be plopped in a vector.
i.e.,
[:map [:items [:vector :string]]]
;; the input I receive is
{:items ["a" "b" ...]}
;; or, sometimes, when there is only one item
{:items "a"}
;; my desired output from m/decode is that :items should always be a vector
{:items ["a"]}#2023-03-1008:25Bingen Galartza IparragirreYou can use :tuple instead of :vector
The docs say: A :tuple describes a fixed length Clojure vector of heterogeneous elements#2023-03-1008:26CaseyYes, but :items isn't fixed length it can have length >=1#2023-03-1008:28Bingen Galartza IparragirreUps, I didn't read the example properly 😅#2023-03-1008:32CaseySome sort of transformer might work, but I don't want to apply this to every instance of string/:vector in a schema#2023-03-1008:44ikitommiI would:
(def vectorify-transformer
(let [coders {:vector #(cond-> % (not (vector? %)) (vector))}]
(mt/transformer
{:decoders coders
:encoders coders})))
(m/decode [:map [:items [:vector :string]]] {:items ["a"]} vectorify-transformer)
; => {:items ["a"]}
(m/decode [:map [:items [:vector :string]]] {:items "a"} vectorify-transformer)
; => {:items ["a"]}#2023-03-1008:46ikitommithere is mt/collection-transformer which ensures the correct type, but doesn’t create collections of non-collection items. There could be a similar built-in for this too.#2023-03-1008:46CaseyIs there a way to apply a transformer to just a specific key in the schema?#2023-03-1008:46ikitommisure, many ways, simplest being:
[:vector {:encode/string vectorize} ...]#2023-03-1008:47CaseyAh.. in the past i've always defined my own simple schema#2023-03-1008:47Caseym/-simple-schema that is#2023-03-1008:48ikitommi… or you could tag the keys and read those in the transformer:
[:map [:items [:vector {:vectorize true} :string]]]
or
[:map [:items {:vectorize true} [:vector :string]]]#2023-03-1008:49ikitomminon need for simple-schema here, if instance properties are good enough. or a new property for more declarative transformation#2023-03-1008:49Caseyaha, that's a nice solution.. it keeps the schemas as data#2023-03-1008:50CaseyCan the transformer get access to the original schema?#2023-03-1008:51ikitommitransformers can read the schemas (and properties) at transfromer creation time, so it’s also super efficient, e.g. creating a decoder from this would mount the transforming function:
[:map [:items [:vector {:vectorize true} :string]]]
, but for this:
[:map [:items [:vector :string]]]
… the transformer known there is nothing to do and doesn’t emit any code#2023-03-1008:51ikitommiyes#2023-03-1008:51ikitommisee the mt/default-value-transformer which reads the :default key from schema. there is a :compile option to access the schema.#2023-03-1008:52ikitommireturning nil from :compile as “nothing to do here”#2023-03-1008:54Casey:compile should return nil in the nop case, or the encoder/decoder fn itself?#2023-03-1008:57ikitommifrom :compile.#2023-03-1008:59Caseybrilliant!
(def vectorize-transformer
(let [coders {:vector {:compile
(fn [schema _]
(when (some-> schema m/properties (find :vectorize))
#(cond-> % (not (vector? %)) (vector))))}}]
(mt/transformer
{:decoders coders
:encoders coders})))#2023-03-1009:11Caseymany thanks @U055NJ5CC malli is great 🙂 sometimes i find it difficult to find the extension points like that. diving into the core code is a great tip#2023-03-1009:15ikitommi👍#2023-03-1010:43Casey@U055NJ5CC given schema [:vector {:vectorize true} :string] , (m/properties schema) returns {:vectorize true} . What is the equivelent malli.core function to get the :string part? of which there may be more than one in the case of :tuple or :enum#2023-03-1010:44CaseyUsually I use (m/form schema) to get the plain data and then things like (second form) or (nth schema 2)#2023-03-1011:09ikitommiThere is m/type, m/properties and m/children for this#2023-03-1010:15niwinzhello o/
I think I found an inconsistent behavior between clj and cljs, in cljs, the uuid is properly validated using regex, but on CLJ the validation is relied to UUID/fromString, it turns out that fromString accept incomplete UUID's as valid uuids.#2023-03-1010:16niwinzI think a better approach would be to perform regex validation always, what do you think?#2023-03-1010:27niwinzhttps://github.com/metosin/malli/issues/867#2023-03-1011:33ikitommiThanks for reporting, PR to fix this (I think regex is good) would be most welcome.#2023-03-1012:08niwinzok, I will do it#2023-03-1013:15stathissiderisif I have
(def schema1
{::id int?
::country string?})
(def schema3
{::secondary-id ::DOES-NOT-EXIST
::age int?})
(def merged (merge
(malli/default-schemas)
schema1
schema3))
Is there any way to get an error because of trying to refer to the ::DOES-NOT-EXIST recipe? I remember figuring this out before but I didn’t make a note at the time 😕#2023-03-1013:26ikitommiCurrently, the registries do not check the references eagerly to see if everything is linked correctly. m/schema does check those eagerly:
(def schema1
{::id int?
::country string?})
(def schema3
{::secondary-id ::DOES-NOT-EXIST
::age int?})
(mr/set-default-registry!
(merge
(m/default-schemas)
schema1
schema3))
(m/schema [:map ::id ::age])
; => [:map :user/id :user/age]
(m/schema [:map ::id ::secondary-id])
; =throws=> {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :user/DOES-NOT-EXIST}}#2023-03-1013:26ikitommithere could be a helper to verify the correctness of a registry, but is not atm.#2023-03-1013:29stathissideristhanks, I did try this, but I think I’m using incorrectly because it doesn’t work with valid references:
(def schema1
{::id int?
::country string?})
(def schema2
{::secondary-id ::id
::age int?})
(malli/schema
(merge
(malli/default-schemas)
schema1
schema2))
I get an exception#2023-03-1013:34ikitommimalli/schema expects one schema, you have a full registry there.#2023-03-1013:35stathissiderisoh, so I’d have to check every schema one by one, right?#2023-03-1013:35ikitommiyes#2023-03-1013:35ikitommi(m/schema
;; schema syntax
[:map ::id ::secondary-id]
;; options
{:registry (merge
(m/default-schemas)
schema1
schema2)})
; => [:map :user/id :user/secondary-id]#2023-03-1013:36ikitommithat’s correct syntax for m/schema#2023-03-1013:36stathissideristhanks!#2023-03-1013:36ikitommi.. but checks just what is in the schema syntax. to verify all schemas in registry, you have to pull the schemas (via malli.registry/schemas) and check them 1 by 1, I guess.#2023-03-1013:37stathissiderismy use case is to merge registries and check whether they have any broken links, let me see if I can come up with a solution#2023-03-1013:54stathissideriskinda works (will make less imperative):
(defn check-registry [registry]
(let [schemas (-> registry
mr/registry
mr/schemas)]
(doseq [[k _] schemas]
(when (or (string? k) (and (qualified-keyword? k)
(-> k namespace (str/starts-with? "malli") not)))
(malli/schema [:map k] {:registry registry})))))#2023-03-1014:33ikitommiyou could check just your own custom registry, no need to crawl the default schemas I guess. btw, there is m/-reference? to check if the key is a valid reference type (string or qualified kw).#2023-03-1014:35stathissiderisboth great tips, thank you!#2023-03-1014:37stathissiderisI filtered out the malli schemas because the check fails when I pass it one of them#2023-03-1014:40ikitommiyes, there is a bunch of schemas that can’t be created without children, e.g. (m/schema :schema) fails, it requires 1 child, (m/schema [:schema :int]). Same with :map-of, :enum , :or, :vector etc.#2023-03-1014:41ikitommithere is ::m/schema and ::m/val as internal schema types, marking eager references and entry values, both require a child.#2023-03-1014:43stathissiderishmm simply merging my own schemas into a registry (and not the default malli schemas) and checking the result with my function doesn’t work, presumably because they use int? and string? which only exist in the default schemas#2023-03-1015:25ikitommiYes, you have the full registry when validating (via options or global registry), but just need to loop the own ones#2023-03-1013:43ikitommiIdea: a mr/requiring-resolve-registry that can be used to create var-schemas without needing to register those:
;; a simple custom Var-schema
(def Over
(m/-simple-schema
{:type `Over ;; here: use qualified symbol so we reconstruct the schema from form!
:compile (fn [_ [value] _]
{:pred #(and (int? %) (> % value))
:min 1
:max 1})}))
;; normal usage
(m/validate [Over 6] 7)
; => true
;; defauls schema + auto-resolve qualified symbols
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
(mr/requiring-resolve-registry)))
;; as symbol
(m/validate [`Over 6] 7)
; => true
;; as symbol
(m/validate ['user/Over 6] 7)
; => true
;; durable
(-> [Over 6]
(edn/write-string)
(edn/read-string)
(m/validate 7))
;=> true
… schema-like simplicity with durability 💪#2023-03-1013:45ikitommiimplementation for the new registry would be ~7 loc.#2023-03-1106:00jasonjcknthat’s pretty cool , its not for everyone but very legit and simple approach i like it #2023-03-1220:07joshkhi’ve been reading the docs and can’t find an answer to this: can i validate one attribute of a map based on the value of another attribute in that map, e.g. :attr2 must be a hyphenated string of :attr1 which is a vector of strings? many thanks 😎#2023-03-1319:00escherizeyeah here’s an example:#2023-03-1319:00escherizehttps://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D#2023-03-2121:15joshkha very late thank you, @U051GFP2V, this is exactly what i was looking for#2023-03-1221:37niwinzhello, I'm trying to use the malli.dot namespace and I found it a but buggy or at least the feeling is very strange when I use it with custom composite registry (default+ mutable)
user=> (v/schema :app.rpc.commands.files/file)
:app.rpc.commands.files/file
user=> (class (v/schema :app.rpc.commands.files/file))
malli.core$_schema_schema$reify$reify__10396
There you can look that the schema is correctly available on the registry (v/schema ...) is the same as (m/schema ... {:registry v/default-registry})...
If I use it as is, I get an exception:
user=> (md/transform :app.rpc.commands.files/file {:registry v/default-registry})
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
:malli.core/invalid-schema
If I use it wrapped in a schema I get an empty record node:
user=> (md/transform [:schema :app.rpc.commands.files/file] {:registry v/default-registry})
"digraph {\n node [shape=\"record\", style=\"filled\", color=\"#000000\"]\n edge [dir=\"back\", arrowtail=\"none\"]\n \n \n}\n"
On trying passing my composite registry as local registry on schema props, it fails:
user=> (md/transform [:schema {:registry v/default-registry} :app.rpc.commands.files/file])
Execution error (IllegalArgumentException) at malli.core/-property-registry (core.cljc:257).
Don't know how to create ISeq from: malli.registry$composite_registry$reify__8529
And it only works I pass a plain map to local registry (derefing the atom used for mutable registry):
user=> (md/transform [:schema {:registry @v/registry} :app.rpc.commands.files/file])
"digraph {\n node [shape=\"record\", style=\"filled\", color=\"#000000\"]\n edge [dir=\"back\", arrowtail=\"none\"]\n \n \":app.common.validation/set-of-strings\" [label=\"{:app.common.validation/set-of-strings|:app.common.validation/set-of-strings\\l}\", [...] tail=\"odiamond\"]\n}\n"
#2023-03-1221:39niwinzIs this expected behavior, Do I need wrap always on [:schema when I want to use the md/transform? it is expected that I can't pass a registry instance to a local registry and it only accept a plain map type of registries? why I can't pass the registry as options to the md/transform instead of passing the local registry to the [:schema ?#2023-03-1221:40niwinzthe same happens with plantuml#2023-03-1221:47niwinzmy sensation is that the md/transform ignores the registry passed as options#2023-03-1221:55niwinzWould be nice to be able to do this:
(md/transform [:schema {:registry {"File" :app.rpc.commands.files/file}} "File"] {:registry v/default-registry})
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
:malli.core/invalid-schema
This will allow have better, human readable name for the type and not the fully qualified keyword on the dot output#2023-03-1410:06niwinzAfter digging on it I think all is related to the pointer schema, I'm not clearly understand why it works this way:#2023-03-1410:07niwinzI mean if I look up a schema with a keyword using a registry using m/schema it will return a pointer schema that will not show the form and will not walk, I need to explicit m/deref it, but I have no evident way for check if the schema is a pointer or real schema... I expect m/schema return ` real schema and not a pointer but maybe I'm missing something here that is not evident#2023-03-1517:36ikitommimalli.dot is not complete (and platuml is built on top of it), should add that to docs. Any improvements / rewrite of it welcome.#2023-03-1517:39ikitommifor pretty naming, something like a :title property could be used?#2023-03-1709:30niwinzI have my own impl for openapi right now for penpot, and working on custom describe module
I will show you the result when it is finished and we will see if it make sense to contribute it back to malli 😄
I have played with the dot and plantuml to understand how they works but I don't need them on the near future, maybe later#2023-03-1709:48ikitommiown impl to openapi? Describe module? Sounds interesting. Reitit OpenAPI3.1 support just landed on master btw#2023-03-1709:49niwinzoh, I wasn't aware of that#2023-03-1709:49niwinz#2023-03-1709:51niwinzjust a screenshot, I'm right now generating the full openapi.json for the penpot rpc api
this is split in two modules: one that generates the schemas for the body from malli and one custom for generate the base openapi.json structure#2023-03-1709:54niwinzI mean, I have started from swagger code from malli and adapted it to openapi with probably some customizations that I need my self, just a WIP right now, I will look on the openapi support that recently landed to the malli and look on if I can use it as-is#2023-03-1710:26ikitommi👍 Reitit + OpenAPI was merged just few days ago, needs still testing before a release. If you have something that is missing/bad in the official impl, happy to pull them in. Also, the code is split between reitit & schema libs (malli, spec-tools, schema-tools) atm.#2023-03-1710:29niwinzAh, now I understand why I missed it, I;m not ussing reitit, I mean I use it but just for basic routing, behind that we have a classic RPC, this is why I use malli for define schemas for all metadata, and then generate openapi.json using the malli ability to walk the schemas (that is awesome)#2023-03-1710:30niwinzJust a example:#2023-03-1710:31niwinzwe started replacing spec with malli on the codebase, and malli is just awesome#2023-03-1710:45ikitommiI have an old example of doing reitit + malli + rpc, just a sec…#2023-03-1710:46ikitommihttps://github.com/metosin/reitit-example:
• the web app (just looping through the actions and generating endpoints): https://github.com/metosin/reitit-example/blob/master/src/clj/example/backend/routes.clj
• actions: https://github.com/metosin/reitit-example/blob/master/src/clj/example/backend/dispatch.clj#2023-03-1710:51ikitommireally like the command/cqrs/rpc pattern, one can program in the business/domain terms, not with http/rest. a screenshot of an app with cqrs + fsm + malli (served via reitit).#2023-03-1713:04niwinzyep, very similar, we also started with query/mutation pattern
but on the end we normalized all to commands, the semantic difference is that queries start the name with get-#2023-03-1716:33niwinzThis is the candidate output for the describe module:
FileWithPermissions -> Object {
id => UUID,
features => FileFeatures -> Set[String],
hasMediaTrimmed => boolean,
commentThreadSeqn => integer,
name => string,
revn => integer,
modifiedAt => Instant,
isShared => boolean,
projectId => UUID,
createdAt => Instant,
data? => anything,
permissions => Permissions -> Object {
type => keyword,
isOwner => boolean,
isAdmin => boolean,
canEdit => boolean,
canRead => boolean,
isLogged => boolean
}
}#2023-03-1716:34niwinzthat i'm working on right now, it will be used for display a human readable representation of the datatype that malli schema represents#2023-03-1322:25andreThoughts on making malli.util/get and malli.util/get-in schemas like merge, union and select-keys are?
To avoid code duplication, I'm writing schemas like: [:map [:foo (m.u/get domain-schema ::domain-schema/foo)]], would be nice to be able to write [:map [:foo [:get domain-schema ::domain-schema/foo]]] instead.#2023-03-1420:42ikitommiThat would be handy, but you could also add the schemas to registry and just say [:map [:foo ::domain-schema/foo]]#2023-03-1515:29andreBut I'd need to manually register all the keys, right? Something like the below, where register-keys walks over the schema and adds them to registry?
(register-keys [:map [::foo any?]])
#2023-03-1515:39ikitommiYes, something like:
(my-register! ::foo any?)
(my-register! ::bar int?)
[:map ::foo ::bar]
#2023-03-1515:42andreLooks a little error prone, as users would need to remember to register every single key, and we would also need to worry about the same key being registered twice and the last one overwriting the first one. That doesn't happen with :get/:get-in since it looks specifically for one schema.
Would it be ok if I tried opening a PR for :get/:get-in as schemas? Or do you have a strong preference for the registry approach?#2023-03-1422:27cap10morganis it possible to define a schema for semi-homogenous maps? I have a map with a specific (optional) key that needs to point at one schema for its value (when present) but then every other key in the map is of a different type whose values are all of yet another type. to make things extra fun, I need that optional key to go through a custom decoder fn. so the specific key's schema looks like:
[:and
[:map-of ::at-context :any] ; for ::at-context's decode fn
[:map ["@context" {:optional true} ::context]]]
and the schema for the other k-v's in the map is:
[:map-of ::iri ::val]
but I have not found a way to combine those so that a map of mostly {::iri ::val} can have an optional {"@context" ::context} in it.#2023-03-1508:37ikitommiissue and discussion here: https://github.com/metosin/malli/issues/43#2023-03-1508:55ikitommiCould take a stab at merging :map and :map-of (https://github.com/metosin/malli/issues/43), best options so far:
Given data:
{:id #uuid"839a35ba-1be1-43d1-af31-0c8cff06690b"
123 "kikka"
456 "kukka"}
Schema to describe it could be:
1️⃣ ::m/default would validate just the extra keys as a map
[:map
[:id :uuid]
[::m/default [:map-of :int :string]]]
2️⃣ ::m/default would validate all extra key-value pairs via :tuple
[:map
[:id :uuid]
[::m/default [:tuple :int :string]]]
thoughs?#2023-03-1509:26dharriganI think option #1 is clearer to read.#2023-03-1510:11juhoteperiWhat about just [::m/default [:int :string]]? Just "special case" like other :map values.
But OK, I see the value that it is a real Malli schema.#2023-03-1510:15dharriganI think it would be nice to make it explicit that it's either a tuple of values or a map-of values, rather than relying upon special "knowledge" that [:int :string] means a map. However, I'm have not that strong a feeling, and thus I can see the value in just [:int :string] too 🙂#2023-03-1510:30juhoteperiI think option 2 is strange. Because the extra values are some number of those tuples, which :map-of describes. If we want to match regular :map keys drop :tuple.#2023-03-1510:45Ben Sless1 is clearer#2023-03-1515:46ikitommi1 it is then. Draft here https://github.com/metosin/malli/pull/871#2023-03-1519:57ikitommiAbout :map + :map-of + mt/strip-extra-keys-transformer … 🧵#2023-03-1519:57ikitommiby default stripping works like this:
(m/decode
[:map
[:x :boolean]
[:y :int]]
{:x 1, "kikka" "kukka", 3 4}
(mt/strip-extra-keys-transformer))
; => {:x 1}#2023-03-1519:58ikitommidisabled for explixitely open maps:
(m/decode
[:map {:closed false}
[:x :boolean]
[:y :int]]
{:x 1, "kikka" "kukka", 3 4}
(mt/strip-extra-keys-transformer))
; => {:x 1, "kikka" "kukka", 3 4}#2023-03-1519:59ikitommiwhatbout :map + ::m/default?
(m/decode
[:map
[:x :boolean]
[:y :int]
[::m/default [:map-of :int :int]]]
{:x 1, "kikka" "kukka", 3 4}
(mt/strip-extra-keys-transformer))
; 1) => {:x 1, "kikka" "kukka", 3 4}
; 2) => {:x 1, 3 4}#2023-03-1520:01ikitommiI think:
• 1️⃣ is more consistent with current behavior (and easier to implement)
• 2️⃣ sounds what I would expect#2023-03-1520:51escherize"kikka" "kukka" is an extra entry, so I think it should be stripped. so 2️⃣ seems more right#2023-03-1606:40opqdonutif I had an API that had ::m/defaults, I'd be pretty peeved if they got stripped#2023-03-1606:40opqdonutperhaps we need a separate strip-extra-keys-transformer and strip-unknown-keys-transformer#2023-03-1606:41opqdonutI've always kinda understood strip-extra-keys as meaning strip-unknown-keys#2023-03-1607:38ikitommiOne can run apply the mt/strip-extra-keys-transformer in two phases:
1. before the ::m/default values have been transformed, e.g. we don’t know the validity of extra keys/values. With this, I would like to keep all extra keys
2. after ::m/default values have been transformed -> we could validate all (non-explicitly defined key-values) and strip away everything that doesn’t match the schema#2023-03-1607:41ikitommisame numbers (1 & 2) as before, just with more context. For example, in current reitit, the mt/strip-extra-keys-transformer is applied BEFORE the normal transformers, so 2 would not work => the childs are not transformed and thus, all the entries could be invalid.#2023-03-1607:43ikitommiI think this could be handled in mt/strip-extra-keys-transformer:
1. check if the schema has ::m/default key
2. If it has, mount a :leave transformer that validates all the extra key-values and strips away the invalid ones.#2023-03-1607:43ikitommiboils down to what does “unknown” keys mean, ping @US1LTFF6D.#2023-03-1608:11opqdonutyeah exactly#2023-03-1608:12opqdonut> after ::m/default values have been transformed -> we could validate all (non-explicitly defined key-values) and strip away everything that doesn’t match the schema
I think this is the right way to go#2023-03-1610:25ikitommithat is non-trivial to solve. Given schema:
[:map
[:x :int]
[::m/default [:map-of :int :int]]]
For value:
{:x 1, 1 1, 2 2, "3" "3", "4" "4"}
… we can trip the explicit keys easily so it’s:
{1 1, 2 2, "3" "3", "4" "4"}
… but how do we know which entries are valid?
1. validate them 1-by-1 {1 1}, {2 2}, … and merge all valid submaps together. This kinda works but as one can provide any schema as ::m/default, this might not always correct, e.g. given [::m/default [:map-of {:max 2} :int :int]]
2. validate everything once => doesn’t work
3. try all combinations => bad idea
ideas welcome, @US1LTFF6D.#2023-03-1610:57opqdonuthmm#2023-03-1610:57opqdonutdissoc the "known" keys, validate the rest using the default schema#2023-03-1610:57opqdonutso it would fail with {:max 2}#2023-03-1611:00opqdonutI'm not sure I see the problem. What were you thinking of?#2023-03-1611:43ikitommifor extra keys:
{1 1, 2 2, "3" "3", "4" "4"}
correct answer after stripping the invalid keys would be:
{1 1, 2 2}
only way to do that I can think of is the 1) - validate key-values 1-by-1 and merge all valid ones together. This should work for many/most of the cases. but not for all.#2023-03-1611:44ikitommimaybe this is just something to document, “best effort for ::m/default keys”#2023-03-1611:44ikitommiand a option to disable it.#2023-03-1611:46opqdonuthmm#2023-03-1611:46opqdonutI would've expected
(m/decode [:map-of :int :int] {1 1, 2 2, "three" "3"} (mt/strip-extra-keys-transformer))
to be {1 1, 2 2}#2023-03-1611:46opqdonutbut it seems strip-extra-keys-transformer doesn't do anything for map-of?#2023-03-1611:47ikitommicurrently, it does not, noticed the same, I’ll make it work too.#2023-03-1611:48opqdonutyeah, well after that works, can't you just reuse that behaviour for ::m/default ?#2023-03-1611:50opqdonutso (m/decode [:map [:x :int] [::m/default def]] val) would delegate to (m/decode def (dissoc v :x))#2023-03-1707:16ikitommithe -transformer of :map already covered the delegation, so just needed to add the :map-of stripping and it works. I’m surprised how well this played out!
(m/decode
[:map
[:x :int]
[::m/default [:map
[:y :int]
[::m/default [:map-of :int :int]]]]]
{:x 1, :y 2, :z 3, 1 1, "2" 2, 3 "3", "4" "4"}
(mt/strip-extra-keys-transformer))
; => {:x 1, :y 2, 1 1}
https://github.com/metosin/malli/pull/871/commits/a1bb82f03e6e30a2d4b05f0404abdac8e2f58e90#2023-03-1707:19ikitommi… and contructed a :tuple schema of the :map-of so we can iterate over key-values without stuffing those into a map. less garbage and don’t have to worry about :map-of level constraints like :min and :max properties.#2023-03-1707:51opqdonutyeah it's pretty nice#2023-03-1519:57ikitommiAbout :map + :map-of + mt/strip-extra-keys-transformer … 🧵#2023-03-1607:44ikitommianother thread about :map + :map-of parsing 🧵#2023-03-1607:45ikitommiGiven:
(def schema
[:map {:registry {'int [:orn ['int :int]]
'str [:orn ['str :string]]}}
[:id 'int]
["name" 'str]
[::m/default [:map-of 'str 'str]]])
(def valid
{:id 1, "name" "tommi", "kikka" "kukka", "abba" "jabba"})
which would be better:
• 1️⃣ - merge the parse results into a single map
(m/parse schema valid)
;{:id [int 1]
; "name" [str "tommi"]
; [str "kikka"] [str "kukka"]
; [str "abba"] [str "jabba"]}
• 2️⃣ - keep the ::m/default parse results separately
(m/parse schema valid)
;{:id [int 1]
; "name" [str "tommi"]
; :malli.core/default {[str "kikka"] [str "kukka"]
; [str "abba"] [str "jabba"]}}#2023-03-1607:58ikitommidefaulting to 2️⃣ here, without that, one has to reparse the parse-results to find out which entries are from the default. As a side-effect, this disallows :malli.core/default key in the map-instances, but that’s ok IMO.#2023-03-1608:03ikitommiyeah, this is good, thanks duckie:
(def schema
[:map
[:id :int]
["age" :int]
[::m/default [:map-of :string :string]]])
(m/parse schema {:id 1, "age" 13, "kikka" "kukka"})
;{:id 1, "age" 13
; :malli.core/default {"kikka" "kukka"}}#2023-03-1608:13opqdonutI'd have expected 1️⃣#2023-03-1608:14opqdonutI think mostly your processing code would "know" which keys are the defaults#2023-03-1608:15opqdonutcould you get something like 2️⃣ using :and if you really want it?#2023-03-1608:44ikitommi:thinking_face:#2023-03-1707:06ikitommi@US1LTFF6D - 1️⃣ has issues with key collisions, so we have to use 2️⃣. Given:
(def schema
[:map {:registry {'str [:orn ['str :string]]}}
[['str "kikka"] :any]
[::m/default [:map-of 'str 'str]]])
(def value {['str "kikka"] true, "kikka" "kukka"})
with 1️⃣ :
(m/parse schema value)
; => {[str "kikka"] [str "kukka"]}
(m/unparse schema *1)
; => ::m/invalid
with 2️⃣ :
(m/parse schema value)
; => {[str "kikka"] true, :malli.core/default {[str "kikka"] [str "kukka"]}}
(m/unparse schema *1)
; => {[str "kikka"] true, "kikka" "kukka"}
… root cause is that the parse result tuples (returned via miu/-tagged) are implemented as MapEntries, which have equality to a vector of size two and thus can collide with user defined vector keys. Not common, but possible.#2023-03-1707:13opqdonutAren't collisions like that a problem in other places as well?
user=> (m/parse [:or [:orn [:s :string]] [:vector any?]] "s")
[:s "s"]
user=> (m/parse [:or [:orn [:s :string]] [:vector any?]] [:s "s"])
[:s "s"]
#2023-03-1707:15opqdonutuser=> (m/parse [:map-of [:or [:orn [:s :string]] [:vector any?]] :int] {[:s "s"] 1 "s" 2})
{[:s "s"] 2}#2023-03-1707:33ikitommioh, true that. the first one works thou:
(def schema [:or [:orn [:s :string]] [:vector any?]])
(m/parse schema "s") ; => [:s "s"]
(m/unparse schema *1) ; => "s"
(m/parse schema [:s "s"]) ; => [:s "s"]
(m/unparse schema *1) ; => [:s "s"]
#2023-03-1707:34ikitommi.. but the second doesn’t, because they are pushed as keys into the map:
(= (miu/-tagged :s "s") [:s "s"]) ; => true
#2023-03-1707:36ikitommiso, the guideline should be “if you plan to use vector keys in maps, use tag names that don’t clash with your domain data”, e.g.
(m/parse
[:map-of [:or [:orn [`s :string]] [:vector any?]] :int]
{[:s "s"] 1 "s" 2})
; => {[:s "s"] 1, [malli.core-test/s "s"] 2}#2023-03-1707:45ikitommiok, removed the ::m/default wrapping, https://github.com/metosin/malli/pull/871/commits/7f948422c70becbf0fa8772e496f629dc246a643#2023-03-1707:45opqdonutgreat#2023-03-1707:46opqdonutI'm reviewing the PR now#2023-03-1614:30ikitommi::m/default fallback works recursively:
(def schema
[:map
[:x :int]
[::m/default [:map
[:y :int]
[::m/default [:map
[:z :int]
[::m/default [:map-of {:gen/max 4} :int :int]]]]]]])
(json-schema/transform schema)
;{:type "object",
; :additionalProperties {:type "integer"},
; :properties {:x {:type "integer"}
; :y {:type "integer"}
; :z {:type "integer"}},
; :required [:x :y :z]}
(m/explain schema {:x 1, :y "2", :z 3, 42 false})
;{:schema ...,
; :value {:x 1, :y "2", :z 3, 42 false},
; :errors ({:path [:malli.core/default :y]
; :in [:y]
; :schema :int
; :value "2"}
; {:path [:malli.core/default :malli.core/default :malli.core/default 1]
; :in [42]
; :schema :int
; :value false})}
(mg/generate schema)
; => {:x -1, :y 423, :z -1130868, 41388000 -28671, -3869 41788, 38 -20324}#2023-03-1618:33cap10morganit appears that you can't attach decode fns to map keys in a :map schema. is that correct? if so, is there another way to do that? I have been doing [:and [:map-of ::key-schema-with-decode :any] [:map [:specific-key ::val-schema]]] but it's kind of clunky and I'm hoping there's a more idiomatic way to do it.#2023-03-1619:17ikitommiYou can attach decode fns to map keys#2023-03-1619:17cap10morganoh great! I must have been holding it wrong then 🙂#2023-03-1619:21cap10morganI'm testing against your latest commit in https://github.com/metosin/malli/pull/871 and it is working great for my use case!#2023-03-1619:31ikitommigood to hear, should be ready to review tomorrow.#2023-03-1620:43cap10morganhmm... I can't get a decode fn on a :map key to work. it needs to decode the key not the value. is that supported?#2023-03-1620:44cap10morganI have this: [:map ["@context" {:optional true, :decode/edn my/decode-fn} ::context]]#2023-03-1620:45cap10morganmy/decode-fn just checks for :context and returns "@context" instead#2023-03-1620:47cap10morganso I'm trying to get maps like {:context ...} to be decoded into {"@context" ...} by that schema w/ a transformer named :edn#2023-03-1620:59cap10morganI attached the decoder at the higher [:map {:decode/edn ...} ...] and checked for :context in the map itself instead and that works. so I'm good 👍#2023-03-1706:40opqdonutYeah I think the key-munging happens on the :map level. Let me check some old code of mine.#2023-03-1706:41opqdonut(defn key-renamer
"A transformer that renames keys of input maps based on the `column`
properties in the schema. If no `column` property is found,
`default-key-fn` is used (defaults to `identity`)."
([column]
(key-renamer column identity))
([column default-key-fn]
(mt/transformer {:name :key-renamer
:decoders {:map {:compile (fn [schema _]
(let [mapping (key-mapping schema column)]
(mt/-transform-map-keys #(get mapping % (default-key-fn %)))))}}})))#2023-03-1706:41opqdonutThat's what I did for a similar case#2023-03-1706:42opqdonut(deftest test-key-renamer
(let [schema (m/schema [:map
[:our-a {:column "a"} :int]
[:our-b {:column "b"} :int]
["c" :int]
[:our-nested {:column "nested"}
[:map
[:our-a2 {:column "a"} :int]
[:our-b {:column "b2"} :int]]]])]
(is (= {:our-a 1
:our-b 2
:c 3
:unknown 4
:our-nested {:our-a2 5
:our-b 6}}
(m/decode schema
{"a" 1
"b" 2
"c" 3
"unknown" 4
"nested" {"a" 5
"b2" 6}}
(domain/key-renamer :column keyword))))))#2023-03-1708:23ikitommiAdd support for default/fallback branch for :map (https://github.com/metosin/malli/pull/871) merged in master. Test/user reports welcome!#2023-03-1708:25ikitommithis is a nice bonus:
(m/decode
[:map-of :int :int]
{1 1, 2 "2", "3" 3, "4" "4"}
(mt/strip-extra-keys-transformer))
; => {1 1}#2023-03-1716:59escherizemy colleague showed me some interesting behavior with decode. It has to do with what happens when a transformer throws.
minimal example:
(mc/decode :sequential "32" (mtx/string-transformer))
It cropped up during a usage with an :or like this:
(mc/decode [:or :sequential :int] "32" (mtx/string-transformer))
Which you’d expect would return 32, but no. it throws:
clojure.lang.ExceptionInfo: :malli.core/child-error
{:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :sequential, :properties nil, :children nil, :min 1, :max 1}}
at malli.core$_exception.invokeStatic (core.cljc:138)
malli.core$_exception.invoke (core.cljc:136)
~It’s probably best if a transformer can never throw~ 1️⃣~, but at least we need to handle exceptions in the parent schema~ 2️⃣ ~or both~ 3️⃣~.~
Update: Just needed a child for the :sequential schema. the exception is getting thrown on schema creation.#2023-03-1717:55ikitommiI believe it's the schema creation: try (m/schema :sequential) , it requires 1 child, e.g
(m/schema [:sequential :any]). m/decode coerces the data syntax into schema.#2023-03-1718:09escherizeGood call, I should have read the error message more carefully.#2023-03-1721:15danielnealheya, I’m trying to build a custom simple-schema using the example in the docs.
(def Between
(malli/-simple-schema
{:type `Between
:compile (fn [_properties [min max] _options]
(when-not (and (int? min) (int? max))
(malli/-fail! ::invalid-children {:min min, :max max}))
{:pred #(and (int? %) (>= min % max))
:min 2 ;; at least 1 child
:max 2 ;; at most 1 child
:type-properties {:error/fn (fn [error _] (str "should be betweeb " min " and " max ", was " (:value error)))}})}))
However I get a “child error”, when I try to use it
(malli/form [Between 10 20])
:malli.core/child-error
{:type :malli.core/child-error, :message :malli.core/child-error, :data {:type sqa.api-reitit.filter/Between, :properties nil, :children [10 20], :min 0, :max 0}}
It looks like malli.core/-simple-schema is not getting the correct min and max values - it expects :max and :min at the top level of the map passed to it - but I changed that and got some other errors using the schema so there may be some other things I’m missing#2023-03-1721:26danielnealoh, I guess it's bad timing, this has just changed on master but unreleased #2023-03-1800:19fuadCan a local registry be a composite registry?
I'm trying something like
[:map {:registry
(malli.registry/composite-registry
(malli/default-schemas)
(malli.time/schemas))}
...]
and that gives me a
Exception: java.lang.IllegalArgumentException: Don't know how to create ISeq from: malli.registry$composite_registry$reify__6973
#2023-03-1800:20fuadIf I use a plain map it works fine.#2023-03-1807:19ikitommifor now, not. But not sure if there is any good reason for not working.#2023-03-1808:31ikitommiplease check if there is an issue of this, if not, could you make one?#2023-03-1818:19ikitommismall-big release`[metosin/malli "0.10.3"]` to celebrate the 2.5M downloads 🥳
• Add support for default branch ::m/default for :map schema, https://github.com/metosin/malli/pull/871
• mt/strip-extra-keys-transformer works with :map-of
• m/default-schema to get the ::m/default schema from entry schema
• m/explicit-keys to get explicit keys from entry schema (no ::m/default)
• Simplify content-dependent schema creation via new 3-arity :compile function, https://github.com/metosin/malli/pull/866
• FIX Repeated calls to malli.util/assoc-in referencing non-existing maps fail, https://github.com/metosin/malli/issues/874
• Updated dependencies (`borkdude/edamame`)
full details: https://github.com/metosin/malli/blob/master/CHANGELOG.md#0103-2023-03-18#2023-03-1916:29ikitommi… and [metosin/malli "0.10.4"] on top of that, had a extra debug print in malli.swagger causing reitit-build to fail.#2023-03-1917:27ikitommiif you want malli + clj-kondo type-checking for :map-of, go 👍 https://github.com/clj-kondo/clj-kondo/issues/2019#2023-03-2011:00niwinzhey @ikitommi, one question about schema walking, there is a way to know the depth of the tree that I'm walking on the walk-fn? I mean, I experimenting a bit with my own describe module, and I need to know when is the top-level schema or not (with top-level schema, I mean the schema passed to the describe function).
I'm aware that workaround is just attach a specific/internal property to the schema before walking, but I'm just curios if there are another way to do that.#2023-03-2012:27ikitommipath is accumulated into walker:
(m/walk
[:map
[:id :int]
[:address [:map
[:street :string]
[:location [:tuple :double :double]]]]]
(fn [schema path children _]
(prn "path:" path schema)
(m/-set-children schema children)))
;"path:" [:id] :int
;"path:" [:address :street] :string
;"path:" [:address :location 0] :double
;"path:" [:address :location 1] :double
;"path:" [:address :location] [:tuple :double :double]
;"path:" [:address] [:map [:street :string] [:location [:tuple :double :double]]]
;"path:" [] [:map [:id :int] [:address [:map [:street :string] [:location [:tuple :double :double]]]]]#2023-03-2012:27niwinzohh, the path, thanks, completely forgotten about path#2023-03-2012:29ikitommime too 🙂#2023-03-2016:50cap10morganI'm having some trouble figuring out how decoder fns are looked up in custom transformers. I have one that looks like this:
(mt/transformer {:name :foo :decoders {::my-schema (fn [v] ...)}})
...and the fully-qualified ::my-schema is defined in a custom registry that is set in the {:registry my-registry} opts map when I create the coercer that I want to use that decoder fn (and to which I pass my custom transformer). However, the decoder fn is never called. Is this because it's defined in a custom registry? Do I need to pass that custom registry into the transformer itself somehow for it to find it?#2023-03-2018:24niwinzThis do not answers directly your question/issue...
but I think you can find it useful: I'm have the following default transform
(def input-transformer
(let [default-decoder
{:compile (fn [s _registry]
(let [props (m/type-properties s)]
(::decode props)))}
default-encoder
{:compile (fn [s _]
(let [props (m/type-properties s)]
(::encode props)))}
coders {:vector mt/-sequential-or-set->vector
:sequential mt/-sequential-or-set->seq
:set mt/-sequential->set
:tuple mt/-sequential->vector}]
(mt/transformer
{:name :penpot
:default-decoder default-decoder
:default-encoder default-encoder}
{:name :string
:decoders (mt/-string-decoders)
:encoders (mt/-string-encoders)}
{:name :collections
:decoders coders
:encoders coders}
)))
This allows attaching decoders directly to schema type-property (see the ::decode key)
(def! ::set-of-strings
(m/-simple-schema
{:type ::set-of-strings
:pred #(and (set? %) (every? string? %))
:type-properties
{:title "set[type=string]"
:description "Set of Strings"
:error/message "should be an set of strings"
:gen/gen (tgen/set (generator :string))
::oapi/type "array"
::oapi/format "set"
::oapi/items {:type "string"}
::oapi/unique-items true
::decode (fn [v]
(into #{} non-empty-strings-xf (str/split v #"[\s,]+")))}}))#2023-03-2018:26niwinzThe main idea here is: avoid attach custom defined schemas constantly to the transformer, I just allow attach the transformer directly to the schema type-properties#2023-03-2018:36cap10morganyeah I'm doing that in a lot of places too. here I want it in the transformer b/c it needs to close around a value being passed to the transformer "constructor" fn#2023-03-2019:21Michael GardnerI'm getting a hang with generate and :or on 0.10.3, and I'm trying to understand why. This also happens with 0.10.2.#2023-03-2019:22Michael Gardnergenerating with my-long or my-double works fine, with or without props#2023-03-2019:27Michael Gardnerwriting it inline as (m.gen/generate [:or {:min 1} my-long my-double]) works too#2023-03-2019:28Michael Gardnerwell, it ignores the :min which I guess is expected#2023-03-2020:05ikitommi(m/schema [(m/schema :int)])
is a minimal repro. This is a syntax error (you can't use Schema instances in first position of the vector syntax), but this should fail fast, not go to endless loop. Issue welcome!#2023-03-2020:06ikitommibtw, you can use :type-properties for generators:
(def my-long
(m/-simple-schema
{:type :my-long
:pred #(instance? Long %)
:type-properties {:gen/schema :int}}))
(def my-double
(m/-simple-schema
{:type :my-double
:pred #(instance? Double %)
:type-properties {:gen/schema :double}}))
(def long-or-double
(m/schema [:or my-long my-double]))
(mg/generate long-or-double)#2023-03-2020:16Michael Gardnergot it, thanks! My actual use case is with dates, so I need to mess with the min/max props and then fmap from long -> date. Would I have to use :compile if I wanted to use :type-properties for the generators?#2023-03-2020:20ikitommiyou don't have to use :compile to set type-properties, with most functions in malli that return IntoSchema instances, you can pass in :type-properties directly#2023-03-2020:53Michael GardnerI'll have to take some time to digest that. Thank you#2023-03-2105:38ikitommifor dates, did you check malli.experimental.time?#2023-03-2120:05cap10morganI keep running into a need to define something like a combination of :map and :map-of where I can assign specific key schema to specific value schema, but several different ones for the same map. The exact key values aren't known ahead of time, but when they conform to a certain schema, then the value should also conform to its schema. Is there any way to do that?#2023-03-2120:19cap10morganor maybe [:map [::key-schema ::val-schema]] can do that? I may have tricked myself into thinking it needed exact key values on the left. trying it now...#2023-03-2120:23cap10morganhmm... no. looks like it doesn't work. or I'm doing it wrong.#2023-03-2205:29ikitommiDid you check https://github.com/metosin/malli/blob/master/README.md#map-with-default-schemas#2023-03-2213:11cap10morganYeah that was helpful for some of these use cases, but not all. I don’t know any of the exact map keys ahead of time, just can write schemas for them. And then I need to map several of those to their respective value schemas. So :map is too rigid with keys but :map-of only allows one key schema and one value schema.#2023-03-2213:12cap10morganI need multiple key schemas each mapping to their own value schema#2023-03-2213:33cap10morganSomething like [:map-of [:or …]]#2023-03-2214:56ikitommiI would use :or like you suggested, either:
[:map-of [:or ...] ...]
[:or [:map-of ... ...] [:map-of ... ...]]
#2023-03-2215:05cap10morganwould the former work? the latter seems like it would say "either the entire map must conform to this :map-of or this :map-of" etc. rather than what I want which is each key-value pair getting its own key-schema-value-schema pair.#2023-03-2215:18cap10morganand [:map-of [:or ::key1-schema ::key2-schema ...] ...] doesn't work b/c I need to map specific value schemas to specific key schemas. rather than "any of these key schemas can apply and then any of these value schemas can apply"#2023-03-2215:38ikitommiso, this is not what you are looking for?
(def schema
[:map
[:x :int]
[:y :int]
[::m/default [:or
[:map-of :string :string]
[:map-of :boolean :boolean]]]])
(mg/sample schema)
;({:x 0, :y 0}
; {:x -1, :y 0, "G" "6"}
; {:x -1, :y -1}
; {:x -3, :y -3}
; {:x 1, :y -3}
; {:x 2, :y 5}
; {:x 2, :y 1, true true, false true}
; {:x -62, :y -1, "EB6wR" "oF", "i3m7" "5Qv", "18" "", "GgV3" "te10p", "57dU2r3" "", "J4Db" "X1z35"}
; {:y 1,
; "4F" "6L",
; "DZ" "2T0o",
; "N75OTk9" "U82v",
; "65jV3HS0" "pEtxQwqb",
; "tM" "DrWy568x",
; "a1ZmzBji" "o4TS",
; "Meh" "3cjN197",
; :x 1}
; {:x 39, :y -34, true false, false false})
(m/validate schema {:x 1, :y 2, "a" "a"})
; => true
(m/validate schema {:x 1, :y 2, false true})
; => true
(m/validate schema {:x 1, :y 2, false true, "a" "a"})
; => false#2023-03-2215:40cap10morganhmm... that might work. the one wrinkle is I don't always have a static key -> value schema to put into the :map schema. but it might be OK to just have [:map [::m/default [:or [:map-of ::foo ::bar] [:map-of ::baz ::qux]]]]?#2023-03-2215:40cap10morganbut that's a good new thread to pull on, thanks!#2023-03-2215:47cap10morganoh wait, no it isn't what I want. the key thing is your last validate call that mixes the two :map-of defaults. I want to be able to get validate => true for that.#2023-03-2215:48cap10morganI'm trying to say "this map will have keys of schema a that map to values of schema a' and keys of schema b that map to values of schema b' and keys of schema c that map to values of schema c'"#2023-03-2215:48cap10morganI think I just have to use a :fn schema for that for now#2023-03-2216:53cap10morganIf I were to work on implementing this, would this syntax be reasonable?
[:map-of [::a ::a'] [::b ::b'] [::c ::c']]
#2023-03-2217:48ikitommiThat starts to be close to :multi, don't want to add more syntax into :`map-of`. There is also opportunities like https://github.com/metosin/malli/blob/master/src/malli/destructure.cljc#L19-L27#2023-03-2217:50cap10morganOK maybe a whole new schema type then? not sure what to call it. it really feels like a natural extension of :map-of#2023-03-2217:50cap10morgannot sure I follow the destructure stuff?#2023-03-2217:51cap10morgancould also be an extension of :map that takes schemas for keys, but also not sure how to disambiguate that#2023-03-2217:53cap10morganmaybe... like... :map-tuples b/c it kind of consumes the map as key-value pairs?#2023-03-2217:55cap10morganspeaking of... can you already consume a map like that? something like [:or [:tuple ::a ::a'] [:tuple ::b ::b'] [:tuple ::c ::c']]?#2023-03-2306:03ikitommisee the MapLike from the link above, bit similar.#2023-03-2306:03ikitommipasting here too:
[MapLike
[:or
[:tuple [:= :keys] [:vector ident?]]
[:tuple [:= :strs] [:vector ident?]]
[:tuple [:= :syms] [:vector ident?]]
[:tuple [:= :or] [:map-of simple-symbol? any?]]
[:tuple [:= :as] "Symbol"]
[:tuple [:fn -qualified-key?] [:vector ident?]]
[:tuple [:ref "ArgType"] any?]]]
#2023-03-2306:11ikitommiImplementing a new type of Schema into malli core means at least the following:
1. design (discussion on the need for it + syntax)
2. implementation (`malli.core` or other ns)
3. error messages (`malli.error`)
4. transforming (`malli.transform`)
5. generator (`malli.generator`)
6. json-schema mapping (`malli.json-schema`)
7. clj-kondo mappings (`malli.clj-kondo`)
8. optionally working with utils (`malli.util`) like merge, union etc.
9. optionally providers (`malli.provider`)
10. other:
a. tests
b. documentation
c. DCE & code size on CLJS
d. performance#2023-03-2306:15ikitommithe current MapLike is just for validating + parsing, to make it full Schema would require a lot of work, so haven’t done that. If you would like to implement such, I propose you’ll write an Issue of this, let’s discuss there the best option for the 1. on the list, ok?#2023-03-2313:54cap10morganLooks like this is pretty much it, if a little implementation-specific: https://github.com/metosin/malli/issues/881#2023-03-2313:55cap10morganOpened 2 hours ago 😂 #2023-03-2313:56cap10morganAlthough I wonder if that’s calling for and semantics or or#2023-03-2315:10delaguardoor :)#2023-03-2315:16delaguardothis should be something like a merge of two and more homogenous maps.#2023-03-2315:16cap10morgancool, yeah. that's exactly what I'm looking for as well.#2023-03-2315:18cap10morganI also think it would be a very natural extension of :map-of (perhaps activated by using vector tuples inside instead of the current syntax of just [:map-of :key-schema :value-schema]). But it sounds like @U055NJ5CC would prefer a new schema type altogether.#2023-03-2121:34Rob HaisfieldAnyone have a macro for converting types and interfaces from TypeScript into Malli schemas? I want to use TypeScript libraries through the JavaScript interop, but I also want to benefit from the TS safety.#2023-03-2204:44Ben SlessConvert from from ts to json schema and then it's a short distance#2023-03-2204:44Ben SlessThere are packages for the former#2023-03-2205:28ikitommihttps://github.com/metosin/malli/issues/656#2023-04-1614:24Rob HaisfieldWhat is the best path from a TS library to JSON? Specifically I’d love to use the AT Protocol api to do some experiments outside of what Bluesky provides (happy to give invites to anyone helping me get to malls schemas on this, btw) https://www.npmjs.com/package/@atproto/api#2023-04-1614:47Rob HaisfieldOkay seems like this library will do it. I have to ask though, if it’s just running TS—>JSON Schema and then using Malli’s decode functionality, why hasn’t it been built yet? Are there any gotchas? https://github.com/YousefED/typescript-json-schema#2023-04-1614:49Ben SlessFrom personal experience I can share that typsescript to json schema works poorly and doesn't cover all possible forms#2023-04-1614:50Ben Slesshad better luck with ts-json-schema-generator but it was still iffy#2023-04-1614:50Ben SlessIf anyone has any experience with TS and can just dump the AST as JSON I'd rather process it in Clojure#2023-04-1614:59Rob HaisfieldDo you remember in what ways it was iffy?#2023-04-1614:59Ben SlessBecause it couldn't emit certain type declarations#2023-04-1614:59Ben Slessso I just got "object"#2023-04-1614:59Rob HaisfieldI’d also be interested in the AST JSON approach you described but that sounds less straightforward for a relative beginner to both TS and CLJS like me to make #2023-04-1615:00Ben SlessOnce you do that CLJS is irrelevant. In theory we just need to pop the hood of one of the existing emitters and start hacking#2023-04-2016:06Rob Haisfield@U04V15CAJ I’m curious your thoughts on this? Seems like you were asked about this at Dutch Clojure Days#2023-04-2016:07borkdudeCan you summarize your question so I don't have to read this whole thread?#2023-04-2018:34Rob HaisfieldWhat do you think about some sort of automatic conversion from TypeScript libraries to Malli schemas? I was thinking that TS->JSON Schema->Malli Schema could work, but seems like there are some gotchas at the TS->JSON Schema step#2023-04-2018:35borkdudeNo special thoughts on that, I haven't done much TS so far#2023-03-2306:11ikitommiImplementing a new type of Schema into malli core means at least the following:
1. design (discussion on the need for it + syntax)
2. implementation (`malli.core` or other ns)
3. error messages (`malli.error`)
4. transforming (`malli.transform`)
5. generator (`malli.generator`)
6. json-schema mapping (`malli.json-schema`)
7. clj-kondo mappings (`malli.clj-kondo`)
8. optionally working with utils (`malli.util`) like merge, union etc.
9. optionally providers (`malli.provider`)
10. other:
a. tests
b. documentation
c. DCE & code size on CLJS
d. performance#2023-03-2219:23cap10morganStill curious if anyone can help me out w/ attaching custom encoders / decoders to schema in the transformer? https://clojurians.slack.com/archives/CLDK6MFMK/p1679331018221749
I need to do it in the transformer and not in the schema def b/c it has to close over a value I'm passing to a fn that creates the transformer. But my ::custom-schema encoders / decoders are never called. Built-in things like :string work, but not my custom ones.
How does encoder / decoder dispatch work w/ the map keys you pass to a transformer in the :encoders / :decoders values?#2023-03-2219:50cap10morganworked around this by writing a different function that merges my closure value directly into the registry instead. kinda clunky but I can't figure out how the decoder / encoder dispatch works to the :decoders / :encoders maps in the transformer itself for custom schemas.#2023-03-2314:26niwinzAfter working a bit with creating custom schemas, I think that would be useful if :gen/gen value can be a delay or function, for initialize the generator lazily, only when it is used. What do you think?#2023-03-2316:05ikitommithat’s a great idea! with function doing requiring-resolve it would really lean.#2023-03-2316:05ikitommi(just importing test.check takes some time)#2023-03-2319:27niwinzthat is not that I have in mind but it also applies#2023-03-2319:28niwinzwith lazy I mean having the ability to defer the creting the generator instance when it is requested#2023-03-2319:31niwinzSomething like this that I have in mind, provide a function that returns the generator, when it is requested, also there is the option to use clojure.core/delay, it caches the value on second use#2023-03-2319:32niwinzbut having the ability to pass a symbol and resolve it once is needed it also a nice to have#2023-03-2319:32niwinzwhat do you think about the "delay" or function ? if it is ok, I can try to make draft PR for it#2023-03-2406:03ikitommiI think function would be a good start.#2023-03-2406:04ikitommithere is m/eval to for optionally plugging into sci, no-op for normal functions#2023-03-2406:04ikitommithis also works btw:
(defrecord My [])
(mr/set-default-registry!
(merge
(m/default-schemas)
{::my (m/-simple-schema
{:pred any?
:type ::my
:type-properties {:gen/schema ::my-map
:gen/fmap map->My}})
::my-map [:map [:x :int] [:y :int]]}))
(mg/sample ::my)
;(#{:x 0, :y 0}
; #{:x -1, :y 0}
; #{:x 0, :y 0}
; #{:x 0, :y 0}
; #{:x 0, :y 1}
; #{:x 7, :y 0}
; #{:x -6, :y -1}
; #{:x 2, :y -7}
; #{:x -2, :y -4}
; #{:x 36, :y -1})#2023-03-2316:10cap10morganWhat's the correct way to use mu/update-properties on a schema in a registry? When I try just merging the return value back into the registry under the same key, I get invalid-schema exceptions.#2023-03-2316:13cap10morganfor example:
(-> registry
(update ::fqh/commit mu/update-properties assoc :encode/edn
(partial encode-commit-edn parsed-context))
(update ::expanded-iri mu/update-properties assoc :encode/edn
#(json-ld/expand-iri % parsed-context))))#2023-03-2719:13ikitommiif you are constructing the registry and try to reference schemas that are not yet registered, it’s an error.#2023-03-2719:14ikitommiyou can only reference things already in the registry.#2023-03-2719:15ikitommiI would use https://github.com/metosin/malli#mutable-registry and add the schemas imperative one-by-one. Just like with spec.#2023-04-0413:59cap10morganThe schemas are already in the registry. I'm just trying to update their properties. But the returned value from mu/update-properties is invalid.#2023-04-0413:59cap10morganbut perhaps a mutable registry is still the right answer there#2023-03-2408:07hifumi123Is it possible to use malli to validate a JS object against a schema? Or do I always need to run js->clj beforehand?#2023-03-2411:36juhoteperiThere is no way to validate JS objects, but you could use decoding to coerce JS object to clj maps:
(defn to-map [^js o]
(into {} (map (fn [[k v]]
[(keyword k) v])
(js/Object.entries o))))
(js/console.log
(pr-str
(m/decode [:map
{:decode/json to-map}
[:a :string]
[:b :string]
[:c [:map {:decode/json to-map}
[:foo :string]]]]
#js {:a "a"
:b "b"
:c #js {:foo "bar"}}
(mt/json-transformer))))#2023-03-2411:21Yab MasI'm seeing some really weird behaviour when trying to keywordize the result of my db-call (which is done with firestore-clj)
The below structure is what I want, but it only keywordizes the top-level keys.
(let [board-link (get-board-link
{:platform "jira"
:board-id 10001})]
(m/decode board-link-schema board-link (mt/key-transformer {:decode keyword})))
Now the weird thing. The below example does behave as expected and keywordizes all sub-maps. This is 100% a copy of the first example with only the board-link binding being substituted with the result of the function-call in the first example (copied from the repl). This doesn't make any sense to me. Is it possible that the db-call adds something to the result that confuses malli?
(let [board-link {"item-links" [{"jira-item-id" "EX-6", "monday-item-id" 4176528394}],
"jira" {"board-id" 10001},
"monday" {"board-id" 3990111892}}]
(m/decode board-link-schema board-link (mt/key-transformer {:decode keyword})))
#2023-03-2411:26juhoteperiYes, it is possible that the db is returning some objects that look like maps, but arent.#2023-03-2411:26juhoteperihttps://github.com/lurodrigo/firestore-clj/blob/master/src/clj/firestore_clj/core.clj#2023-03-2411:26juhoteperiThe code seems to refer to some DocumentSnapshot class for example#2023-03-2411:27juhoteperiand there is ds->plain function#2023-03-2411:28juhoteperiYou could call class for one of the map-like items from the db call#2023-03-2411:36Yab MasMja, the ->plain is for the real-time-data but. I'm doing a normal pull, about which the docs say it returns a normal map: You can use pull to fetch the results as a map.#2023-03-2411:37Yab MasWhich is inline with what I see:#2023-03-2411:37juhoteperiIt was working with the top-level keys? What about objects inside item-links?#2023-03-2411:39juhoteperiOr the values of jira and monday properties#2023-03-2411:40Yab Mas#2023-03-2411:40juhoteperiBut what is the class for those values#2023-03-2411:43Yab MasOk I see, java.util.HashMap#2023-03-2411:43Yab MasNot sure what the best solution is, but should be able to figure it out from here. Thanks 🙏#2023-03-2411:44juhoteperiYou can add :decode/json (fn [v] (to-clj-map v)) to the map schemas where the value need to be convereted from HashMap to clj#2023-03-2411:44juhoteperi(into {} v) or something might be enough#2023-03-2411:44Yab MasAh, perfect. Will try.#2023-03-2411:45juhoteperiOnce it is clj map, Malli will handle key and value transformations inside the map like regular#2023-03-2413:27Yab MasJust confirming this is now solved. Still had to tinker a bit, but that was more because of me being relatively new with malli. Thanks again for the quick help 😉#2023-03-2412:53souenzzoHello. On this code:
(m/explain [:map {:closed true}]
{:a 43})
=>
{:schema [:map {:closed true}],
:value {:a 43},
:errors ({:path [:a], :in [:a], :schema [:map {:closed true}], :value nil, :type :malli.core/extra-key})}
It reports :value nil. it is expected? I was expecting it to report :value 43#2023-03-2413:10ikitommiit could retain the value. PR welcome#2023-03-2413:26souenzzoActually it should be something with my custom schema registry.#2023-03-2620:05MariusHi, I’d like to create a schema with all keys optional from an existing schema, but the following code is throwing exception :malli.core/invalid-schema … Is is enough to call it like this or what did I miss?
def employee-schema [:map ...]
def registry {
::employee employee-schema
::employee-all-optional (mu/optional-keys employee-schema)
}#2023-03-2622:17pppaulsometimes malli wants a real scheme object. I am not sure if this is the problem, but it's worth a try in this case. I think I only use schema objects for my registry, but I also mainly do low level schemas for that. i have a feeling that this employee type schema is probably better being outside the registry#2023-03-2719:11ikitommi(def employee-schema [:map [:x :int]])
(def registry
(merge
(m/default-schemas)
{::employee [:map [:x :int]]
::employee-all-optional (mu/optional-keys employee-schema)}))
(m/deref ::employee {:registry registry})
; => [:map [:x :int]]
(m/deref ::employee-all-optional {:registry registry})
; => [:map [:x {:optional true} :int]]#2023-03-2719:11ikitommimost utilities take either data or Schema instance, marked as ?schema in the function arguments. Those, which accept just Schema, are marked as schema#2023-03-2818:56MariusThanks a lot for your help!
I could pinpoint the error to the following line:
(def employee-schema [:map
...
[:employee-type ::employee-type]
...
])
::employee-type is defined in the registry
(def registry
(merge
(mc/default-schemas)
{::employee-type (into [:enum] (map :code data/employee-types))
::employee employee-schema}))
but I assume that I also have to take that outside as a def employee-type ?#2023-03-2716:04Braden ShepherdsonI'm using Malli schemas for intelligent, deep conversion from JS to CLJS data and back. it works pretty well overall. but I ran into an issue with :multi schemas. I'm trying to use a property (`:type`) to differentiate between a few similar maps, but of course it's not as simple as :dispatch :type if the input might be a JS object which hasn't been converted to a CLJS map yet.
is there a more clever way to do this than making the dispatch function :dispatch #?(:clj :type :cljs #(if (object? %) (keyword (.-type %)) (:type %)))?#2023-03-2716:06Braden ShepherdsonI'm playing with
(mut/merge
[:map
[:type [:enum :foo :bar]] ...shared stuff]
[:multi {:dispatch :type} [:foo [:map ...]] ...])
now#2023-03-2716:08Braden Shepherdsonbut that appears to have the same issue.#2023-03-2719:00ikitommiI can’t figure out a more cleaver way to do that than you proposed.#2023-03-2716:19JoelIs there a way to combine local/custom registry (if that’s the right terms)? Admittedly this is a naive attempt…
[:map {:registry (merge mc/default-registry {::id int?
::country string?}}
::id
[:name string?]
[::country {:optional true}]]
I was planning to modify the default as well (eg. add :pos-int). TBH, I haven’t figured out how I actually want this to work, but trying to understand my options better.#2023-03-2716:56andreI think the correct syntax would be
[::id ::id]
[::country {:optional true } ::country]#2023-03-2716:57Joeli just copied that from the malli site… really asking about the registry part of it.#2023-03-2717:00andreNot sure if that's what you are looking for, but here is what worked for me when I was playing with registries:
(def ^:private schema-registry (atom {}))
(m.registry/set-default-registry!
(m.registry/composite-registry
(m/default-schemas)
(m.util/schemas)
(m.registry/mutable-registry schema-registry)))
#2023-03-2717:03JoelMaybe a bit more direct way i should have asked… Can I mix a custom registry with a local?
(merge mc/default-registry {::id int?
::country string?}}
When I tried this I got an error. I suspect the local map needs to be converted to an IntoSchema one somehow.#2023-03-2718:57ikitommi@UH13Y2FSA currently, the local registry must be data, doesn’t accept IntoSchema instances. This both is a) unfortunate b) slows m/form a lot. Please write an issue, I could figure out how to fix/change this.#2023-03-2806:55opqdonutI ran into a really weird thing with m/walk and m/schema-walker when working on support for recursive schemas in malli.json-schema. Any thoughts? See
https://github.com/metosin/malli/pull/883/commits/1abb34333c57c0b9f723494ab66bd1dcac146fdf#2023-03-2806:57opqdonutThe root cause here is the -walk implementation of -schema-schema#2023-03-2806:57opqdonut(-walk [this walker path options]
(when (-accept walker this path options)
(if (or (not id) ((-boolean-fn (::walk-schema-refs options false)) id))
(-outer walker this path (-inner-indexed walker path children options) options)
(-outer walker this path [id] options))))#2023-03-2806:58opqdonutthis combined with schema-walker basically does (-set-children schema [id]), which changes the child from :int to "Foo" in the test in my commit#2023-03-2807:00opqdonut... and we can avoid this by setting ::walk-schema-refs#2023-03-2811:37opqdonutok this is really weird:
user> (def closed (mu/closed-schema [:schema {:registry {"Foo" [:map [:a :int]]}} "Foo"]))
#'user/closed
user> (m/form closed)
[:schema {:registry {"Foo" [:map [:a :int]]}} "Foo"]
user> (-> closed m/deref m/deref m/form)
[:map {:closed true} [:a :int]]#2023-03-2811:38opqdonutwe've changed the "Foo" , but the change is hidden in m/form, which only shows the id "Foo"#2023-03-2811:38opqdonutand (obviously?) the change isn't propagated to the registry#2023-03-2812:20opqdonutwrote up an issue https://github.com/metosin/malli/issues/884#2023-03-2812:21opqdonutI now have a workaround in PR 883, but weird output from mu/closed-schema persists.#2023-03-2913:41simonaccaHi all,
wondering how to check that an object is of a certain type with Malli.
This certainly works, but is there a more idiomatic way?
(deftype Car [])
(def schema (malli/-simple-schema {:pred #(-> % type (= Car))}))
(malli/validate schema (Car. ))
#2023-03-2914:23delaguardo[:fn #(instance? Car %)]#2023-03-2914:33simonaccaThanks!#2023-03-3012:22MariusIs there a conventient way to recursively make all keys optional using mu/optional-keys ?#2023-03-3012:31opqdonutI think you'll just have to use m/walk to do it#2023-03-3012:31opqdonutThere could be a function for it in malli.util . It comes up often enough.#2023-03-3012:31opqdonutI'm pretty sure I've written it at least once, but can't find it.#2023-03-3012:33opqdonut(defn make-all-keys-optional
"Recursively apply mu/optional-keys to map schemas"
[schema]
(m/walk
schema
(m/schema-walker
(fn [schema]
(if (= :map (m/type schema))
(mu/optional-keys schema)
schema)))))#2023-03-3012:33opqdonutfound it 🙂#2023-03-3012:57MariusAwesome! Thank you very much!#2023-03-3015:02MariusHi all, I have a question regarding Reitit Malli request coercion: When receiving a Transit request, the string decoders defined in my schema are executed, if it is a JSON request, it is only transformed by Muuntaja, but the Malli string decoders seems to be skipped… does that make sense?
Background: I’m using the string decoders to convert strings into dates or big decimals.#2023-03-3019:38MariusFound the solution:
I had to add an additional :decode/json property on my LocalDate / BigDecimal type, which a copy of the :decode/string property.#2023-03-3019:39MariusLooks like this for LocalDate:
(def LocalDate
(mc/-simple-schema
{:type 'LocalDate
:pred #(instance? java.time.LocalDate %)
:type-properties {:error/message "invalid date"
:decode/string #(t/date %)
:decode/json #(t/date %)
:json-schema {:type "string"
:format "date"}
:gen/gen (gen/fmap #(t/>> (t/today) (t/new-period % :days)) (gen/choose -365 365))}}))#2023-03-3019:40Mariust is tick.core#2023-04-0216:33ikitommi👍#2023-03-3022:06simonaccaHi all, one more question: is it possible to walk a spec + value in Malli?
Say for example from the following spec
[:schema {:registry {::id :int
::address [:map
[::street :string]
[::city ::id]]
::user [:map
[:name ::id]
::address]}}
::user]
and example data
{:name -10257,
:my-ns/address
#:my-ns{:street "bhvZWkA2KL37u7y25O3R", :city 122054499}}
one wants to extract all values corresponding to an ::id key, that is: [122054499, -10257]
Can this be done? I look at value transformations but they seem to be hardcoded for a prewalk, while here we need a postwalk. Am I missing something? :thinking_face:#2023-03-3022:40simonaccaDuh, a transformer's encode operation can take an interceptor as an operator for a certain schema.
Or a :compile key if you also need to access the schema, like in this case 🎉#2023-04-0216:32ikitommiDo you have a working code example how to do this?#2023-04-0216:32ikitommijust added a this tip/recipe: https://github.com/metosin/malli/blob/master/docs/tips.md#accessing-both-schema-and-value-in-transformation#2023-04-0401:01simonaccaThanks!#2023-04-0214:00roklenarcicI see there’s a function called -instrument at the end of malli.core, what does it do? How do you instrument your functions?#2023-04-0215:27ikitommiHere's the guide: https://cljdoc.org/d/metosin/malli/0.10.4/doc/function-schemas#2023-04-0501:41steveb8nQuestion: could a defn schema automatically create a table of options like described in this blog post? https://martinklepsch.org/posts/writing-awesome-docstrings.html#2023-04-0501:43steveb8nsince the schema has a :description key it seems like one of the Malli defn macros might be able to do this (or already does so?)#2023-04-1614:02ikitommiPlumatic had something like this, spec might have something like this too?#2023-04-1614:04ikitommiI think it is a good idea. For simple schemas, this would be great, for large/complex ones - not sure. Could malli.experimental.describe be used here? or just pretty m/form? or …. a table? how would a table work with vector-of-maybe-self-recursive-multi schema?#2023-04-1706:03steveb8nYeah not sure if IDEs will understand macro generated docs though. E.g. cursive#2023-04-1706:04steveb8nI was thinking only for kwargs, not for anything more complex#2023-04-1706:19ikitommiif it’s just documenting kwarg names, it should be fine. if types, it’s a rabbit hole, .e.g. optional :registry with type of <<huge schema in english here>>.#2023-04-1706:20steveb8nMy thoughts exactly#2023-04-1706:21ikitommisimple way to play with this would be to add a new function (var) that would be responsible for generating the docstring. By default, it would return the current text as-is, but one could use alter-var-root! the var and swap something custom there. Just pass enough data to the function, maybe:
(defn -create-docstring [docstring _schema _options] docstring)
#2023-04-1707:25steveb8ndo you mean to play and see if the IDE will understand it? I’ve already tried the table style docstrings and Cursive doesn’t understand them. @U0567Q30W am I right that Cursive doesn’t grok docstring tables?#2023-04-1707:25steveb8nstill useful for cljdoc and other tools but my primary use case is IDE docstring features#2023-04-0507:09bitkiller.42Hi, my goal is malli and kondo showing me warnings like this
https://github.com/metosin/malli/blob/master/docs/img/defn-schema.png
Using: Expo/ReactNative (Clojurescript), shadow-cljs 2.22.9 and malli 0.10.4
I've followed the guide on https://cljdoc.org/d/metosin/malli/0.10.4/doc/clojurescript-function-instrumentation
shadow-cljs.edn
{:target :react-native
:modules {:app {:entries []}}
:init-fn
:output-dir "app"
:devtools {:reload-strategy :full
:preloads [re-frisk-remote.preload
kickmanager.dev-preload
malli.dev.cljs-kondo-preload]}
devpreload.cljs_
(ns kickmanager.dev-preload
{:dev/always true}
(:require
[]
[malli.instrument.cljs :as mi]
[malli.dev.cljs :as dev]
[malli.dev.pretty :as pretty]))
(dev/start! {:report (pretty/reporter)})
;;(mi/collect!)
;;(mi/instrument!)
When I instrument the function directly it works partially:
(def plus1
(m/-instrument
{:schema [:=> [:cat :int] [:int {:max 6}]]}
(fn [x] (inc x))))
Using the other ways like
(defn minus
{:malli/schema [:=> [:cat :int] [:int {:min 6}]]}
[x]
(dec x))
I get Schema error when instrumenting function: . TypeError: undefined is not a function when instrument!
(m/function-schemas) is nil, so I also called (mi/collect!) directly but the function-schemas are still not created and then instrument fails with the error above.
Any tips or ideas? I only want some schema checking for functions and data that has kondo/ide support, are there other ways to achieve this? Thx.#2023-04-1614:01ikitommiplease write an issue. not sure what is happening in here.#2023-04-0601:23rafaeldelboniHello folks, I want to create a test fixture for my service to enable malli instrumentation, what is the best approach?#2023-04-0601:24rafaeldelboniCurrently I have this:
(defn with-malli-intrumentation
"Wraps f ensuring there has malli collect and instrument started before running it"
[f]
(mi/collect! {:ns (all-ns)})
(mi/instrument! {:report (pretty/thrower)})
(f)
(mi/unstrument!))
;then I add this to my test ns
(use-fixtures :once with-malli-intrumentation)
#2023-04-0601:24rafaeldelboniThe problem is that the start and stop prints a bunch of ..instrumented nss and ..unstrumented, which makes hard to find the test errors on the repl, one way to this would be wrap instrument! with with-out-str#2023-04-0601:26rafaeldelbonibut it feels wrong#2023-04-0601:27rafaeldelboniI would love to see other folks approach to this#2023-04-1613:58ikitommiprinting should be reduced/removed with https://github.com/metosin/malli/pull/842#2023-04-1613:59ikitommiI think a malli.test helper ns would be good, could have a fixture for this.#2023-04-1913:17rafaeldelbonithat would be incredible convenient#2023-04-0609:02flowthingGiven:
[:map
[:foo int?]
[:bar boolean?]]
I need to specify that :foo is required only if :bar is false. Is content-dependent schemas (https://github.com/metosin/malli#content-dependent-simple-schema) the tool for the job? Is there an example of how to do that available, by any chance?#2023-04-0609:09juhoteperi"Content dependent schema" doesn't mean the value content, but the schema form contents.
Like :map schema takes the key value tuples.#2023-04-0609:10juhoteperiYou could make :foo optional and then add extra fn at the map level to check the map contents and check that if bar is false, foo is required.#2023-04-0609:11juhoteperiBy design one property can't depend on another, so such checks should be at the level where both properties are available -> the map level.#2023-04-0609:12juhoteperi[:and
[:map
[:foo {:optional true} :int]
[:bar :boolean]]
[;fn (fn [v] (if (:bar v) true (some? (:foo v)))]]
https://github.com/metosin/malli#fn-schemas#2023-04-0609:33Ben SlessMulti which dispatches on bar#2023-04-0609:34flowthingThanks!#2023-04-0610:34juhoteperiYeah multi could be an option also#2023-04-0610:35juhoteperiIf you had multiple non-related rules on the map, multi could be inconvenient, but with :and you can have many rules which are independent.#2023-04-0610:42flowthingYeah, I like :and for this, not least because it’s the same solution that I’d use with spec. Not sure why it didn’t occur to me in the first place. 🙂#2023-04-0615:41Ben SlessIirc multi behaves better when generating values if you care about that#2023-04-0610:02LoicHi,
How to represent a zero arity function in malli schema?
[:=> [:cat :int] :int] ;; f accept one int and returns a int
[:=> ? :int] ;; what to put here so f accept no arg and returns a int?
Thank you in advance#2023-04-0610:10delaguardojust [:cat] should work#2023-04-0610:18LoicIndeed, thank you.#2023-04-1613:57ikitommigolfing: :cat should work too.#2023-04-0614:22Andrew LaiI came across a Malli behavior I didn’t understand and wanted some advice.
When I chain two mu/assoc-in calls, it seems like the resultant schema has a duplicated field (see below, new-field shows up twice). Is this expected behavior? Am I misusing the assoc-in function?
(def Address
[:map
[:street string?]
[:city string?]
[:zip int?]
[:lonlat [:tuple double? double?]]])
(def Address2
(-> Address
(mu/assoc-in [:new-field :v1] [:map [:foo :int]])
(mu/assoc-in [:new-field :v2] [:map [:foo :int]])))
Address2
;; => [:map [:street string?]
;; [:city string?]
;; [:zip int?]
;; [:lonlat [:tuple double? double?]]
;; [:new-field [:map [:v1 [:map [:foo :int]]]]]
;; [:new-field [:map [:v1 [:map [:foo :int]]]
;; [:v2 [:map [:foo :int]]]]]]
#2023-04-1114:29opqdonutthat looks like a bug. could you add it to github issues?#2023-04-1116:47Andrew LaiCreated. Thank you for the advice!
https://github.com/metosin/malli/issues/891#2023-04-0710:49armedHi, our project codebase is scattered between several libraries, and we are using :malli function instrumentation extensively. I started getting the following error when I start instrumentation:
clojure.lang.ExceptionInfo: :malli.core/child-error {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :vector, :properties nil, :children nil, :min 1, :max 1}}
it seems that somewhere in one of the libraries :malli/schema metadata or m/=> contains incorrect schema. Error stack trace does not contain useful information about source of the exception. What is a good way to debug this?#2023-04-1613:56ikitommiThat doesn’t look very helpful. Please write an issue - it could report the var-name at least.#2023-04-1616:04armed@U055NJ5CC https://github.com/metosin/malli/issues/896#2023-04-1021:33victor ruotoloHi everyone, I'm trying to define a custom transformer (not sure if necessary, but felt like what we needed) in 3 different ways. Only one of them works, and I don't really understand why the others don't, could anyone help me understand it a bit more?
(def something-schema [:map
[:something [:or :time/local-date :int]]])
(def transformer1
(mt/transformer
{:name :my-transformer
:default-decoder {:compile (fn [schema _]
(fn [x]
(case (m/form schema)
:time/local-date
(try (-> (time-format/formatter "MM/DD/YYYY")
(time-format/parse x)
(time-coerce/to-long))
(catch Exception _ x))
:int
(try (Integer/parseInt x)
(catch Exception _ x))
x)))}}))
(def transformer2
(mt/transformer
{:name :my-transformer
:decoders {:int (fn [x]
(try (Integer/parseInt x) (catch Exception _ x)))
:time/local-date (fn [x]
(try (-> (time-format/formatter "MM/DD/YYYY")
(time-format/parse x)
(time-coerce/to-long)) (catch Exception _ x)))}}))
(def transformer3
(mt/transformer
{:name :my-transformer
:default-decoder {:compile (fn [schema _]
(fn [x]
(case (m/form schema)
[:or :time/local-date :int] (try
(Integer/parseInt x)
(catch Exception _
(-> (time-format/formatter "MM/DD/YYYY")
(time-format/parse x)
(time-coerce/to-long))))
x)))}}))
(comment
(m/decode
something-schema
{:something "12/12/2012"}
transformer3))
#2023-04-1021:34victor ruotoloOnly transformer3 works the way I wanted it to, the other two just return the same string in the keyword :something#2023-04-1021:34victor ruotoloWould anyone know why that's the case?#2023-04-1613:50ikitommi@U01FK0L87DJ, root cause is that you are returning longs, not LocalDate instances as your schema defines. Decoding is a process that should always return values that are valid against the schemas.
• in both 1 & 2: :or selects the first branch that returns valid value. Here, the LocalDate transformation is applied, but an int is returned -> :or ignores that branch as “invalid” and goes to check the next branch. None of the branches match, so original is returned.
• how to fix?
◦ describe the field type as :int instead of :time/local-date
◦ mark it somehow as “can be decoded from string”, e.g. via a custom property
◦ something like:
(def something-schema
[:map
[:something [:int {::type :time/local-date}]]])
(def transformer1
(mt/transformer
{:name :my-transformer
:default-decoder {:compile (fn [schema _]
(when (= :time/local-date (::type (m/properties schema)))
(fn [x] (try ...
(catch Exception _ x)))))}}))#2023-04-1713:03victor ruotoloOoooooh I see, that makes sense. Thank you so much!#2023-04-1121:07Noah BogartIs it possible to have a schema that is a specific value? [:map [:specific "foo"]] would require {:specific "foo"}#2023-04-1121:22Ben Sless[:= value]#2023-04-1121:39Noah BogartOh yes, thank you#2023-04-1121:10DrLjótssonReplace ”foo” with [:enum "foo"]#2023-04-1318:04Anne Maialeis there any way to use malli / generators to give me an example of all the different shapes that can adhere to a schema?
like if I have a map schema with a couple different optional keys and a couple required ones, I'd like to make sure I haven't missed a test case#2023-04-1319:44Ben SlessI doubt that's possible, especially with recursive and sequence schemas the search space is infinite#2023-04-1319:45Ben SlessYou could for a subset of schemas say they can be explored exhaustively then just collect a sufficient number of samples until all options are explored
But the search algorithm isn't tuned for that AFAIK, but for searching failing cases fast#2023-04-1613:18ikitommiI would use generative testing here. Something like:
(require '[clojure.test.check.clojure-test :refer [defspec]])
(require '[clojure.test.check.properties :refer [for-all]])
(require '[malli.generator :as mg])
;; my test-fn
(defn my-fn [{:keys [a b c d]}]
(+ a b c d))
(defspec my-test 100 ;; try 100 times
(for-all [x (mg/generator [:map
[:a :int]
[:b :int]
[:c {:optional true} :int]
[:d {:optional true} :int]])]
(int? (try (my-fn x) (catch Exception _)))))
;; smallest failing case
;{:a 0,
; :b 0,
; :c 0}#2023-04-1613:20ikitommiand as Ben said, there is currently no exhaustive way to do that.#2023-04-1614:59Ben SlessAnd the matter of exhaustive checks is very subtle. You can say "I just care about the shape of data", then you can certainly define a subset of schemas that let you know you have enumerated all their shapes. But it's actually worse, even knowing something is an integer provides little guarantee wrt correctness. What if your algorithm breaks if it's larger than 2^32?#2023-04-1700:17Anne MaialeVery interesting. In terms of the values themselves, I'm going to provide those so I don't actually care what it spits out for the values. But great point, this is a non recursive schema so I hadn't considered that#2023-04-1704:08Ben SlessExample of the sort of bugs I was referring to with value ranges:
https://scribe.rip/m/global-identity-2?redirectUrl=https%3A%2F%2Fthebittheories.com%2Fthe-curious-case-of-binary-search-the-famous-bug-that-remained-undetected-for-20-years-973e89fc212
It's an interesting anecdote, if nothing else 🙃#2023-04-1400:56steveb8nQ: is there a way to apply a defn schema to an anonymous fn? I have a higher order fn that accepts fn args and I want to apply a schema to the incoming fns (which can be anon)#2023-04-1400:57steveb8nI don’t think instrumenting the HOF is applied to the incoming fn args#2023-04-1402:26steveb8nsame for return values which are fns from a HOF. I guess that anon fns cannot be instrumented because instrumentation requires a fn var?#2023-04-1612:58ikitommiyou can instrument functions too.#2023-04-1612:59ikitommifrom https://github.com/metosin/malli/blob/master/docs/function-schemas.md:
(def pow
(m/-instrument
{:schema [:=> [:cat :int] [:int {:max 6}]]}
(fn [x] (* x x))))#2023-04-1613:06ikitommibut, in Clojure, there is no way to check the function arities, argument or return types - but we can use generative testing to brute force the inputs to see if they are of correct type.#2023-04-1613:06ikitommi(def =>plus
(m/schema
[:=> [:cat :int :int] :int]
{::m/function-checker mg/function-checker}))
(m/validate =>plus plus)
; => true
(m/validate =>plus str)
; => false#2023-04-1613:07ikitommi(m/explain =>plus str)
;{:schema [:=> [:cat :int :int] :int],
; :value #object[clojure.core$str],
; :errors ({:path [],
; :in [],
; :schema [:=> [:cat :int :int] :int],
; :value #object[clojure.core$str],
; :check {:total-nodes-visited 0,
; :depth 0,
; :pass? false,
; :result false,
; :result-data nil,
; :time-shrinking-ms 1,
; :smallest [(0 0)],
; :malli.generator/explain-output {:schema :int,
; :value "00",
; :errors ({:path []
; :in []
; :schema :int
; :value "00"})}}})}#2023-04-1613:07ikitommialso, not sure if there is an option to enable the generative function testing into or malli.instrument. If not, PR most welcome!#2023-04-1613:08ikitommiside-note: only non-side effecting functions should be tested.#2023-04-1706:02steveb8nThanks. I found the -instrument method which will work#2023-04-2014:37Craig BrozefskyI saw that java.uti.regex.Pattern was listed as a class Schema, but I do not know how to use it in a schema. I have tried several ways, none working -- after importing. (m/schema 'Pattern) or (m/schema Pattern) ... fail#2023-04-2014:52delaguardodo you want to validate a string against regex?#2023-04-2018:33Craig Brozefskyno, I want to have the value be a Pattern#2023-04-2018:34Craig BrozefskyI have a config file, I have a regex reader macro, that calls re-pattern. I want the schema to express that that key in the map shouldhave a Pattern in it#2023-04-2019:54eskosI've solved a similar issue https://github.com/esuomi/muotti/blob/master/src/main/clj/muotti/malli.clj#L188-L192, basically it's custom simple schema for the instance? function.#2023-04-2020:14RyanWhats a good way to write a schema for a map that has required keys if a certain value (e.g. :type) is a specific value#2023-04-2020:28escherizeSounds like a https://malli.io/?value=%5B%7B%3Atype%20%3Asized%2C%20%3Asize%2010%7D%0A%20%7B%3Atype%20%3Ahuman%2C%20%3Aname%20%22tiina%22%2C%20%3Aaddress%20%7B%3Astreet%20%22kikka%22%7D%7D%5D&schema=%5B%3Avector%0A%20%5B%3Amulti%0A%20%20%7B%3Adispatch%20%3Atype%7D%0A%20%20%5B%3Asized%20%5B%3Amap%20%5B%3Atype%20%5B%3A%3D%20%3Asized%5D%5D%20%5B%3Asize%20int%3F%5D%5D%5D%0A%20%20%5B%3Ahuman%0A%20%20%20%5B%3Amap%0A%20%20%20%20%5B%3Atype%20%5B%3A%3D%20%3Ahuman%5D%5D%0A%20%20%20%20%5B%3Aname%20string%3F%5D%0A%20%20%20%20%5B%3Aaddress%20%5B%3Amap%20%5B%3Astreet%20string%3F%5D%5D%5D%5D%5D%5D%5D.#2023-04-2020:29RyanOh yeah, that’s totally it!#2023-04-2020:29RyanThanks @U051GFP2V!#2023-04-2020:15Ryane.g. a map needs a :date inst if its :type is :appointment#2023-04-2119:21rafaeldelboniHey in a system using sierra's https://github.com/stuartsierra/component, should I call set-default-registry! in a component start lifecycle?
Where/how do you folks change the default-registry?#2023-04-2119:32hifumi123As early as possible. I find or make a root namespace that is loaded before anything else using malli. This assumes you'll set it once and never again, which is always the case in my applications#2023-04-2119:33rafaeldelboniYeah, I do prefer the immutable registry option 🙂#2023-04-2120:51Ben SlessWhy not make the registry a component and inject that?#2023-04-2121:34steveb8nOne possible reason could be defn schemas. They need the registry as a static reference when loading. I do this by having a global (not ideal) reference to the registry#2023-04-2121:36steveb8nFwiw I believe that the registry option is one of the best features of Malli. It allows me to support customer registrys in a multi tenant system#2023-04-2611:58fuadI came across that questions a few weeks ago when I started using malli more heavily. At first I defined the registry as a component, but that meant I couldn't def schemas anymore, since the registry wouldn't be properly set at load time.
In the end I decided to just def the registry itself in a namespace and explicitly pass it to coerce and validate calls in the options map. I am not super happy with this setup because I always need to remember to pass along the registry, which is especially annoying when playing in the repl (I could set the default registry when playing in the repl 🤷).
I could also pass it in the options of every schema I define, which is a bit more verbose.
The last alternative I found is what @U0479UCF48H suggested: have a namespace be the first one to load (e.g. first require in main and user namespaces).#2023-04-2612:00fuadI'm considering adopting the load time one but curious to hear if there's some other alternative, or maybe some way to deal with the downsides of having it as a component.#2023-04-2612:01rafaeldelboniYeah currently I'm doing the same Fuad, I'm handling my custom schemas as I would handle plumatic/schemas
This is one of my studies using this
https://github.com/parenthesin/microservice-boilerplate-malli/blob/main/src/microservice_boilerplate/schemas#2023-04-2612:01rafaeldelboniBut for types like time and other stuff more basic, I was considering use the registry#2023-04-2612:10fuadYeah my only use case for deviating from the default registry right now is the malli.experimental.time stuff.#2023-04-2612:11fuadI guess I could reference the schemas directly instead of using the keywords and that would be more convenient#2023-04-2615:53fuadAlso, not sure if you're using reitit @UMMMKKADU, but I had https://clojurians.slack.com/archives/C7YF1SBT3/p1681752973306179 a few weeks ago. Still haven't figured it out.#2023-04-2617:31rafaeldelboniI'm using those experimental time types, but in the "wrong" way (schema like)
https://github.com/parenthesin/microservice-boilerplate-malli/blob/main/src/microservice_boilerplate/schemas/types.clj#L26-L30#2023-04-2617:34rafaeldelbonihave you tried something different from :time/instant?#2023-04-2617:35rafaeldelboniI remember when trying those having problems specifically with instants#2023-04-2219:53jayIs it possible to use default-value-transformer with recursive schemas? This causes a StackOverflowException
(m/decode [:schema {:registry {::test
[:map
[:x {:default 3} int?]
[:y {:default {:x 5}} [:ref ::test]]]}}
::test]
{}
(mt/default-value-transformer))
which makes sense, but i’m not sure if there’s a workaround?#2023-04-2312:12ikitommithat is an infinitely expanding schema anyways, no value is valid against it.#2023-04-2312:13ikitommi(mg/generate
[:schema {:registry {::test
[:map
[:x {:default 3} int?]
[:y {:default {:x 5}} [:ref ::test]]]}}
::test])
; =throws=> Cannot generate values due to infinitely expanding schema: [:map [:x {:default 3} int?] [:y {:default {:x 5}} [:ref :user/test]]]#2023-04-2312:14ikitommimaking the key optional (or wrapping the value with :maybe) helps:
(mg/sample
[:schema {:registry {::test
[:map
[:x {:default 3} int?]
[:y {:default {:x 5}, :optional true} [:ref ::test]]]}}
::test])
;({:x -1, :y {:x 0}}
; {:x 0, :y {:x -1}}
; {:x -2, :y {:x -1}}
; {:x 1}
; {:x -2}
; {:x 3}
; {:x 0, :y {:x -1, :y {:x -1}}}
; {:x 2, :y {:x -1}}
; {:x -115}
; {:x -1})#2023-04-2312:16ikitommiby default, mt/default-value-transformer does not fill optional values:
(m/decode
[:schema {:registry {::test
[:map
[:x {:default 3} int?]
[:y {:default {:x 5}, :optional true} [:ref ::test]]]}}
::test]
{}
(mt/default-value-transformer))
; => {:x 3}#2023-04-2312:16ikitommiWhen adding also optional keys, it overflows as your example:
(m/decode
[:schema {:registry {::test
[:map
[:x {:default 3} int?]
[:y {:default {:x 5}, :optional true} [:ref ::test]]]}}
::test]
{}
(mt/default-value-transformer {::mt/add-optional-keys true}))
; =throws=> StackOverflowError#2023-04-2312:17ikitommie.g. don’t set default to recursive fields - setting default value is eager operation, no way to stop that.#2023-04-2312:17ikitommihope this helps.#2023-04-2318:51jaythank you for confirming that. figured there wouldn’t be a way to stop the eager operation,#2023-04-2613:15fuadHello! I have a scenario where malli.transform/strip-extra-keys-transformer is behaving in a way that seems unintuitive to me and I'd like to understand it a bit better#2023-04-2613:18fuadMinimal repro case:
(def schema [:map-of :string :int])
(malli/coerce schema {"foo" 1 "bar" :baz} malli.transform/strip-extra-keys-transformer) ; => {"foo" 1}
Initially I expected the coercion to fail because the key itself (`"bar"`) conforms to the schema but the value doesn't.
But I guess the implementation treats map-of by evaluating each map entry tuple as a whole and striping them if they don't conform as a whole.#2023-04-2613:19fuadNow I don't know if there's a right or wrong way to think about this, maybe both perspectives are valid but I'd like to verify my understanding of how coercion of map-of is supposed to behave.#2023-04-2613:37opqdonutI think there's a discussion of this in a ticket#2023-04-2613:37opqdonutjust a sec#2023-04-2613:39opqdonutI guess it was implemented in this PR but I can't see the discussion I'm remembering#2023-04-2613:39opqdonuthttps://github.com/metosin/malli/pull/871#2023-04-2613:39opqdonutPerhaps it was me&Tommi in a chat somewhere...#2023-04-2613:42opqdonutthis is the commit https://github.com/metosin/malli/commit/0e5b5e3ac56479f75c812c1320ac1f1ebfba8f76#2023-04-2613:43opqdonutanyway, what you're seeing is the intended behaviour: all entries that don't validate are stripped#2023-04-2613:43fuadThanks for all the context!#2023-04-2613:45fuadGiven that it is the intended behavior, and I think that's a perfectly valid choice, what would be the way to opt-out of it? I would guess a custom transformer, right?#2023-04-2613:45opqdonutYeah that's certainly one way#2023-04-2613:46opqdonutI'm thinking about some way to override it case-by-case but I didn't figure out anything yet#2023-04-2815:35ikitommiI think there are at least 3 good options:
1. per schema override, e.g. ::mt/strip-extra-keys false property
2. per transformer, like 1 but as a option for mt/strip-extra-keys-transformer
3. both of 1 & 2#2023-04-2620:49Noah BogartIs there predicate for vars? I have a map of keywords to vars and it feels bad to use :any. I posted an https://github.com/metosin/malli/issues/732 but that's a while ago and it's possible something changed that I missed#2023-04-2715:46roltwould this work ? [:map-of keyword? [:fn var?]]#2023-04-2717:31Noah Bogartoh yeah, interesting, let me try that#2023-05-0115:06Noah BogartThat works great, idk why I didn't think of it. Thank you!#2023-04-2815:16cap10morganI have a question about coercion of :or schemas: If the value already conforms to one of the or'd schemas but it can be coerced to the first one, does it make sense to coerce it to that (as seems to be the current behavior)? Wouldn't it be better to leave it alone? Is there a way to do that? For example, I have a schema that is just [:or :keyword :string] but it keeps coercing all of my strings to keywords even though strings are allowed too.#2023-04-2815:17cap10morganShould I just do a validate check before coercing?#2023-04-2815:18cap10morganAnd actually, I'm not sure I can easily do that in this context. I'm using reitit w/ malli request coercion and I need it to leave my strings alone, but it converts them to keywords.#2023-04-2815:26cap10morganperhaps coercion should short-circuit if the value already validates?#2023-04-2815:26cap10morganprobably some performance implications to all of this#2023-04-2815:28ikitommiOr should already short-circuit on first valid value. It has a (small) performance penalty to validate after decode, but think it's the only way to make it work#2023-04-2815:31cap10morganyeah, I guess it's the decode step that's doing the string -> keyword change. I'm using the json-transformer but trying to leave strings as strings. this is not a map key. but the [:or :keyword :string] schema seems to decode the string to a keyword. unless I'm misunderstanding...#2023-04-2815:38ikitommiuse [:or :string :keyword]?#2023-04-2815:39cap10morganyeah, I had thought I couldn't b/c I need EDN to stay as keywords when using that. but then I remembered that that won't activate the json-transformer in the first place. so that might be the key!#2023-04-2815:44cap10morganyeah that seems to work. glad it was a simple PEBKAC error! thanks!#2023-05-0107:51ilmoIs it possible to validate the form of a keyword somehow (maybe using regular expressions)?#2023-05-0108:15valtteriAt least something like this can be done
(def schema [:and
:keyword
[:fn #(re-matches #"[a-zA-Z](\d+)" (name %))]])
(m/validate schema :poni)
;; => false
(m/validate schema :a123)
;; => true
(m/validate schema "a123")
;; => false#2023-05-0205:37Ben SlessSomething I'm missing with generators is schemas algebra, there are scenarios where I'd want to mix a static schema with a potentially dynamic one and generate from that. Instead of writing ad hoc schemas or combinators it would be nice if it were already part of schemas, either in the algebra level, or that generators will also provide how to gen/bind with them (e.g. and)
Thinking about quickcheck with malli#2023-05-0206:30ikitommiInteresting. How would you define the dynamic part?#2023-05-0206:31ikitommithere is :gen/schema which is currently just a value. supporting a function (fn [schema options] …) there would allow one to access the current schema.#2023-05-0206:46Ben SlessIn the context of quickcheck, the problem starts with needing to generate data based on the current state.
You could write your generators ad-hoc and compose those, but what I want is to use the existing state as a "kernel", then provide it to a generator to fill-out the missing bits.
Case example: we simulate a flow of user creation and log in. The correct implementation must generate a log in request for already existing user IDs (logical join?)
Currently I'd need to create a generator that generates user request, samples and id, and overrides it in the request.
What I want is [:and [:map [:id id]] UserLogIn]#2023-05-0215:30escherizeI’m curious to learn more about recursive generators with refs and :malli.core/potentially-recursive-seqex. Besides https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L186-L281 is there more info about limitations / tricks for creating them?#2023-05-0215:52escherizegathering some more info, there’s this thread too: https://clojurians-log.clojureverse.org/malli/2022-12-04#2023-05-0215:56escherizeWoah, This is super interesting too: https://github.com/metosin/malli/pull/792/files#diff-b81a0b70baac8a767d8abe402c8292a9caca1476d4b80d7fa9c6587fdfe659deR1-R3#2023-05-0216:31escherizeDid this https://github.com/metosin/malli/pull/792/files#diff-b81a0b70baac8a767d8abe402c8292a9caca1476d4b80d7fa9c6587fdfe659deR1-R4 namespace ever get turned into a library?#2023-05-0216:31escherizeDid this https://github.com/metosin/malli/pull/792/files#diff-b81a0b70baac8a767d8abe402c8292a9caca1476d4b80d7fa9c6587fdfe659deR1-R4 namespace ever get turned into a library?#2023-05-0414:15SamRepost from #beginners!#2023-05-0415:03Ben SlessMy best advice is avoid fn schema at all cost. There's always a better way to write your schema. If it's very special, write a custom schema
You can also use transformers to massage your data into a form which makes it easy to validate without a fn schema
Write a property based test that makes sure all schemas you generate are valid
Decoders and encoders are your friends, schemas represent ought, not is, let decoders do the heavy lifting for you. String schemas are suspect#2023-05-0419:32SamThank you!#2023-05-0506:46SamAnd to clarify, what I've been doing, verifying the schema at the start of the function, is reasonable?
Is that how people do it?#2023-05-0507:31Ben SlessDepends on your use case, but my suggestion is validate it when data comes in from the outside world into your process, and when it goes out of process (e.g. side effects)
You might want to check it in key interface points inside your program if it's large
But schemas aren't meant to validate all your functions' inputs at runtime#2023-05-0507:31SamSounds good, thanks!#2023-05-0517:19adamfreyIs there a way to add a generator to an existing :malli.core/schema object stored in a var in another namespace?#2023-05-0517:29escherizeYou could alter-var-root to the var, and use u/update-properties to assoc in the generator key of your choice#2023-05-0517:31adamfreym.u/update-properties is what I was loopking for, thank you#2023-05-0517:31escherize(mg/generate
(mu/update-properties [:vector int?] assoc :gen/elements [[3] [4]]))
#2023-05-0517:31escherizegives [3] or [4]#2023-05-0518:47DrLjótssonIs there a way to get pretty printed and concise error messages from failed (reitit) coercion and instrumentation? I have instrumented db queries and coerced api routes that return large amounts of data and it’s difficult to see what the exact offending data is from the exception. If it’s only 1 record it would be great to see only that record, and if it's hundreds, it would be nice to only see a few of them#2023-05-0518:53valtteriDid you try .pretty/exception already? https://cljdoc.org/d/metosin/reitit/0.7.0-alpha3/doc/basics/error-messages#pretty-errors#2023-05-0518:54escherizeI would ask in #C7YF1SBT3 if you havn’t already.#2023-05-0518:59DrLjótssonThanks @U6N4HSMFW , I will look at that! @U051GFP2V , I was also wondering if one can get more legible exceptions from instrumented functions (not reitit-related)#2023-05-0519:01valtteriThere’s similar pretty reporter for malli. I don’t know how it reports instrumented functions but probably quickest way to find out is to try 😉#2023-05-0519:02valtteriAlso since it’s Tommi-code it’s probably extensible#2023-05-0519:30escherizeOh, I know how that works actually, @UGDTSFM4M.#2023-05-0519:30escherizeThere’s a wrapped humanize function you can pass.#2023-05-0519:30escherizeit’s mentioned on the tips and tricks page, iirc#2023-05-0519:31DrLjótssonFor instrumentation?#2023-05-0519:31escherizehttps://github.com/metabase/metabase/blob/master/src/metabase/util/malli.cljc#L50#2023-05-0519:31escherizeyeah for any humanize call#2023-05-0519:32escherizewhat you want to do is plug in your :report function when you call instrument! (or -instument)#2023-05-0519:33escherizewhich is done here: https://github.com/metabase/metabase/blob/master/src/metabase/util/malli.cljc#L69#2023-05-0519:34escherizeand then, you may want to call me/humanize, and plug in some kind of wrap function for it#2023-05-0519:36DrLjótssonOh, wow. This looks amazing. I need to wrap my head around how this can be pieced together but shouldn’t be that hard. #2023-05-0519:37escherizelet me know if u have any more questions#2023-05-0610:46AkizI could use it too, I have a couple of :sequential schemes that return thousands of complex maps and overwhelm Emacs. Isn't there already a similar reporting function? Not to reinvent the wheel 😉 #2023-05-0612:16DrLjótssonI am working on a solution for instrumented functions. Happy to show my solution next week and get feedback!#2023-05-0721:29DrLjótssonSo this is what I managed to achieve. For coercion errors, I added the following custom exception handler to my reitit routes
{:reitit.coercion/response-coercion
(fn [exception request]
(log/error "Response coercion error"
(with-out-str
(pprint/pprint
(malli-error/humanize
(ex-data exception)
:request))))
{:status 500
:body {:message "Response coercion error"
:uri (:uri request)}})
,,,}
This logs the humanized coercion error. Although it prints all failing records in a large dataset, they will probably all follow the same pattern and it's pretty easy to spot what the actual error is.
For instrumentation I added this function as my :report option to instrument!
(defn instrument-error!
[type data]
(let [{:keys [input args output value arity]} data
[cause humanized] (cond
input ["Invalid input" (malli-error/humanize (m/explain input args))]
output ["Invalid output" (malli-error/humanize (m/explain output value))]
arity ["Invalid arity" "Too few or too many args"])]
(throw (ex-info
(str cause "\n" (with-out-str (pprint/pprint humanized)))
{:type type :data data}))))
(I added arity errors for completeness but I guess that Clojure will catch arity errors)
When developing in the REPL, I get the instrumentation error printed as the exception message, which is handy. I also added this to my reitit custom exception handling
(defn instrument-error
[exception request]
(log/error (ex-message exception))
{:status 500
:body {:message "Application error"
:uri (:uri request)}})
;; Custom exception handling map
{,,,
::m/invalid-output instrument-error
::m/invalid-input instrument-error
,,,}
So I get instrumentation errors in a request logged.
I am sure there are better ways to do this so any feedback is welcome. Also, it would be great if the instrumentation error message could include the name of the failing function. I can't locate it in the data received by the :report function but I guess that could be extracted from the stack trace? I have no idea how to do that though.#2023-06-0112:58AkizIt looks good to me, I haven’t come up with anything better.#2023-06-0113:03DrLjótssonAny idea on how to get the name of the instrumented function that failed?#2023-06-0207:48AkizIt is part of the data, you should see it in the exception info#2023-06-0207:49AkizSo something like this for example:
(defn instrument-error!
[type data]
(let [{:keys [input args output value arity]} data
[cause humanized] (cond
input ["Invalid input" (malli-error/humanize (m/explain input args))]
output ["Invalid output" (malli-error/humanize (m/explain output value))]
arity ["Invalid arity" "Too few or too many args"])]
(throw (ex-info
(str cause " in " (:fn-name data) "\n" (with-out-str (pprint/pprint humanized)))
{:type type :data data}))))#2023-05-0915:53cap10morganWhen I already have a validation exception but want to get the humanized error in the catch, this works but is a bit cumbersome: (-> error Throwable->map :data :data :explain me/humanize). Is there a better way or would it be good to add a utility fn like me/humanize-exception to malli for this? Otherwise it's tempting to run the value through validation all over again but w/ m/explain instead but that's kind of wasteful when the exception already contains the explain data.#2023-05-0916:18escherizewhere does your validation exception come from?#2023-05-0916:18escherizemx/defn?#2023-05-0916:19cap10morganThrown by m/coerce#2023-05-0916:22escherizeI usually use ex-data for that:
(try (mc/coerce :int "three")
(catch Exception e (ex-data e)))
;; {:type :malli.core/invalid-input,
;; :message :malli.core/invalid-input,
;; :data
;; {:value "three",
;; :schema :int,
;; :explain {:schema :int, :value "three", :errors ({:path [], :in [], :schema :int, :value "three"})}}}#2023-05-0916:23cap10morganyeah, that's true. that cleans it up a bit 🙂#2023-05-0916:23cap10morgan(-> error ex-data :data :explain me/humanize) then#2023-05-0916:23escherize👍#2023-05-0916:24escherizeLooks not too bad to me#2023-05-0916:24escherizeyou can even plug in a wrap function on humanize right there#2023-05-0916:24cap10morganyeah. if that were documented it wouldn't be so bad. so maybe just a README PR#2023-05-1015:40cap10morganhttps://github.com/metosin/malli/pull/901#2023-05-1014:33cap10morganWould it make sense for a :map schema w/ keyword keys like [:map [:foo :string] [:bar :string]] decoding w/ the json-transformer when it encounters a map like {"foo" "whatever" "bar" "thingy"} to just turn those keys into keywords similar to what e.g. [:map-of :keyword :string] would do? It doesn't seem to do this today, but I'm wondering if a PR to enable it to would be welcome (or to find out that I'm doing it wrong).#2023-05-1014:50escherizeGood question: I would encourage using a key-transformer with the json-transformer. Something like#2023-05-1014:50escherize#_:clj-kondo/ignore ;;nocommit
(require '[malli.core :as mc] '[malli.transform :as mtx] '[cheshire.core :as json])
(mc/decode [:map [:k :string]]
(json/decode "{\"k\": \"v\"}")
(mtx/transformer
(mtx/json-transformer)
(mtx/key-transformer {:decode keyword})))
;; => {:k "v"}#2023-05-1014:51cap10morganwell, I can't b/c I don't necessarily want all keys transformed to keywords. just the ones that are spec'd as keywords in the schema#2023-05-1014:53Noah Bogart[:foo :string] is speccing foo as a keyword#2023-05-1014:54Noah Bogart(mc/validate
(mc/schema [:map [:foo :string]])
{"foo" "a"})
;; => false
#2023-05-1014:54Noah Bogart(mc/validate
(mc/schema [:map ['foo :string]])
{'foo "a"})
;; => true
#2023-05-1014:55escherizeTrue, however in my example above if I add a schema with a string as the key, it would still keywordize it#2023-05-1014:55Noah Bogartoops, you're correct, i misunderstood#2023-05-1014:56escherizeHmm. yeah, it’s tricky#2023-05-1014:57cap10morganyeah these schemas are for an internal representation that often comes in via an http api as json and we're using malli to coerce those to the internal representation#2023-05-1014:58cap10morganbut some keys need to stay strings, so the key-transformer is a bit all-or-nothing#2023-05-1014:58cap10morganwhereas the map schema already implicitly defines those types#2023-05-1014:58escherizeyou could use a modified keyword schema#2023-05-1014:59escherizeon 2nd thought: maybe not for map keys, though#2023-05-1014:59cap10morganthat's what I'm doing currently via an additional :map-of schema in an :and w/ the :map#2023-05-1014:59cap10morganbut it's kind of clunky#2023-05-1014:59cap10morganand the :map already contains all the info it needs to do this#2023-05-1014:59cap10morganhence my question#2023-05-1015:12Noah Bogartlooking at the source, seems the issue is that -json-decoders doesn't do anything special for maps#2023-05-1015:12Noah Bogartvarious other types do transformations: string to keyword, etc#2023-05-1015:13escherizethis is dumb:#2023-05-1015:13escherize(defn m->m-decode-kws [map-schema]
(let [kws (set (map name (filter keyword? (mut/keys map-schema))))
kw-kws-fn (fn [m] (into {} (for [[k v] m] [(if (kws k) (keyword k) k) v])))]
(mut/update-properties map-schema assoc :decode/mine kw-kws-fn)))
(mc/decode (m->m-decode-kws [:map [:k :int] ["j" :int]])
{"k" 1, "j" 2}
(mtx/transformer {:name "mine"}))
;; => {:k 1, "j" 2}#2023-05-1015:13Noah Bogartand json-transformer only accepts map-of special stuff#2023-05-1015:13escherizebut it works#2023-05-1015:13escherizeit records which keys are keywords in kws, then calls keyword on them from the map’s custom transformer. verysweat_g#2023-05-1015:15escherizeI think the behavior could be rightly considered a bug though#2023-05-1015:15cap10morganYeah I agree#2023-05-1015:15cap10morganOr at least an unimplemented feature#2023-05-1015:18Noah Bogarti think raising an issue on github is a good first step#2023-05-1015:48cap10morganhttps://github.com/metosin/malli/issues/902#2023-05-1015:57cap10morganAnd now I realize you could make the exact same case for other literal value schemas like :enum or :=#2023-05-1015:57cap10morganbut that's not an argument against 🙂#2023-05-1217:41escherizefwiw, enum does decode its own keys:
(m/decode [:enum :k] "k" mt/json-transformer)
;;=> :k
#2023-05-1217:42cap10morganOh! Interesting!#2023-05-1015:05cch1Hi all, I’m new to malli but have some experience with spec tools - I’m committed to the switch from spec-tools. I’m struggling with the documentation around immutable registries -here’s my latest puzzle:
(malli/decoder [:map {:registry malli/default-registry}] malli.transform/json-transformer)
That blows up with a cryptic error message deep in the bowels of malli. Remove the reference to the default registry and it works as expected. Likewise, providing the registry as an option to decoder instead of inline in the spec works. I confess to being surprised by the malli registry and schema construction operations. An immutable registry is a big chunk of the value prop (along with transforms) of malli but I can’t master simple things like supplying a registry to transforms.#2023-05-1015:48delaguardotry [:schema {:registry default-registry} [:map ,,,]]#2023-05-1015:51cch1(m/decoder [:schema {:registry m/default-registry} [:map]] mt/json-transformer) gives me the same error.#2023-05-1016:14delaguardocould you add the error?#2023-05-1017:24cch1gats> (m/decoder [:schema {:registry m/default-registry} [:map]] mt/json-transformer)
Execution error (IllegalArgumentException) at malli.core/-property-registry (core.cljc:257).
Don't know how to create ISeq from: malli.registry$custom_default_registry$reify__28151
gats> #2023-05-1114:16cch1I remain stumped and I’ll raise an issue for this. There seems to be a lot of undocumented behavior, perhaps even bugs, around supplying immutable registries when transforming.#2023-05-1217:14escherizeWill this work for you?
(m/decoder :map {:registry m/default-registry} (mt/json-transformer))#2023-05-1217:36cch1Yes, that does work. But it’s strange that it’s possible to associate a registry with a map schema (clearly required for recursive structures) but not have it be generally supported. Or documented as such. In any case, I can change my approach to not associate registries with maps even with the map entries are spec’d in the schema. Instead, I’ll provide the registry everywhere else (decode, decoder, encode, encoder, explain …)#2023-05-1114:41cch1This doesn’t seem right:
(m/decode decimal? 3 (mt/json-transformer))
; ; => 3
I would expect 3M.#2023-05-1222:05p-himikWhat would be the right way to create a spec for a vector where the first element is a keyword and the second one is an int, and the rest of the elements could be anything?
Plain :cat doesn't cut it since it'll validate any sequences.
Using [:and vector? [:cat ...]] should work for validation but doesn't work for generation.
Of course, I can provide a custom generator but that's cumbersome.
Maybe there's some other way? Maybe :cat has some sort of a parameter to constrain the sequence type?#2023-05-1222:15escherizeIs it, basically:
(mg/generate [:cat :keyword :int [:* :string]])#2023-05-1222:16escherizeoh but you need it to be a vector, right. Not sure if that is a thing#2023-05-1222:18escherizeI’d probably do
(mg/generate [:cat {:gen/fmap (fn [t] (vec t))}
:keyword
:int
[:* :string]])
;; => [:SK+3O32 14 ""]#2023-05-1222:20p-himikAh, not that cumbersome actually, especially since I can even go with {:gen/fmap vec}. Thanks!#2023-05-1222:44escherizeCame back to say that 🙂#2023-05-1415:35aaron51Can I use a single malli schema to check both a map and the schema of a json-encoded string within it?#2023-05-1423:31aaron51https://github.com/metosin/malli#to-and-from-json
I think this is close — I’m trying to validate a reitit param that’s JSON encoded without decoding it first #2023-05-1621:53aaron51I asked chatgpt 😂 and it suggested either a simple-schema or a transformer for this purpose. Any thoughts? I’d want all errors in the json schema to bubble up.
What I want to write (in reitit+malli) is:
{:get {:parameters {:query {:q [:json [:map [:value [:and int? pos?]]]]}}}}#2023-05-1621:55aaron51An example URL might be
”/someurl?q=%7B%22x%22:123456789%7D"
(That’s a json map in the query param)#2023-05-1623:10p-himikHow would validating a serialized data structure work without deserializing it first?#2023-05-1700:42aaron51I imagined that I could write an extension/schema/transformer that would deserialize it when it sees the :json keyword.
Maybe I should be looking at coercers? I just don’t know where to extend malli to do this#2023-05-1706:00opqdonutif you want to get the errors in the validation phase, coercers/decoders are the wrong way to go. they don't signal errors#2023-05-1706:02opqdonutit should be relatively straightforward to write a schema that does something like....
(reify
IntoSchema
(-into-schema [parent properties children options]
(reify Schema
(-validator [_]
(let [child-validator (-validator (first children))]
(fn [val]
(child-validator (json-decode val))))))))#2023-05-1706:03opqdonutsomething like that could go into malli itself if you factor out the json-decode function. it could be called something like a lens-schema ...#2023-05-1621:53aaron51I asked chatgpt 😂 and it suggested either a simple-schema or a transformer for this purpose. Any thoughts? I’d want all errors in the json schema to bubble up.
What I want to write (in reitit+malli) is:
{:get {:parameters {:query {:q [:json [:map [:value [:and int? pos?]]]]}}}}#2023-05-1511:43roklenarcicIs there a library that would instrument functions with malli schemas in style of Guardrails, but malli instead of spec?#2023-05-1511:55delaguardoyou mean inline schema definition?
https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-inline-schemas#2023-05-1512:00roklenarcicthat still requires that I run a watch with instrument#2023-05-1512:00roklenarcicdoesn’t it?#2023-05-1512:02roklenarcicand the syntax is a bit nasty, it will break the Cursive parsing of the function#2023-05-1512:02Noah BogartIf you put m/instrument! somewhere in your app, then all calls to mx/defn will instrument automatically#2023-05-1512:03roklenarcicaha#2023-05-1512:03delaguardothere are three different options for the syntax 🙂 probably at least one would be good for Cursive
yes, I think you have to add instrument! somewhere.#2023-05-1512:03roklenarcicwhere would I learn about the syntaxes?#2023-05-1512:03roklenarcicI only see one in docs#2023-05-1512:04delaguardoI usually add it to user.clj that inside of dev source paths. This way it is automatically enabled in my REPL but will not impact production artifact#2023-05-1512:04delaguardohttps://github.com/metosin/malli/blob/master/docs/function-schemas.md#defn-schemas#2023-05-1512:04delaguardomx/defn is the third option#2023-05-1512:05roklenarcicright, but it only seems to support plumata type of syntax#2023-05-1512:07roklenarcicI’ll ask in cursive channel if I can make this work somehow#2023-05-1512:07delaguardohttps://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schema-metadata
how about this? it is still "inlined" but not as plumatic schema#2023-05-1512:08delaguardolooks very close to guardrails to me#2023-05-1512:10roklenarcicthat one requires:
• running m/collect! once I have loaded all namespaces (`collect!` doesn’t work)
• running collect! and instrument! after every change, or running a watch
This was what I initially tried to use, and it’s the same bad pattern as with spec instrumenting via orchestra, that is why I asked for Guardrails style#2023-05-1512:10roklenarcic(mi/collect!)
Syntax error (ClassNotFoundException) compiling at (/private/var/folders/fm/5mzhclpd7mj0tzjjq796ftc00000gn/T/form-init2758266297015755142.clj:1:1).
roklenarcic.myns.company#2023-05-1512:12delaguardoWasn't obvious at first what do you mean by "guardrails style". Looks like exception originated in your code so I would fix it first before looking for another "silver bullet"#2023-05-1512:15delaguardobecause it is ClassNotFound I would assume you don't load all your namespaces before attempt to collect!#2023-05-1512:15roklenarcicIt does not originate in my code.
• I started REPL (without user.clj in the project)
• navigated to empty namespace, with just empty (ns …) form, loaded the namespace
• (require '[malli.instrument :as mi])
• (mi/collect!)#2023-05-1512:16delaguardocould you share full stacktrace?#2023-05-1512:16roklenarcic#error{:cause "roklenarcic.edavki.currency",
:via [{:type clojure.lang.Compiler$CompilerException,
:message "Syntax error compiling at (/private/var/folders/fm/5mzhclpd7mj0tzjjq796ftc00000gn/T/form-init7469228970531054215.clj:1:1).",
:data #:clojure.error{:phase :compile-syntax-check,
:line 1,
:column 1,
:source "/private/var/folders/fm/5mzhclpd7mj0tzjjq796ftc00000gn/T/form-init7469228970531054215.clj"},
:at [clojure.lang.Compiler analyze "Compiler.java" 6825]}
{:type java.lang.ClassNotFoundException,
:message "roklenarcic.edavki.currency",
:at [java.net.URLClassLoader findClass "URLClassLoader.java" 445]}],
:trace [[java.net.URLClassLoader findClass "URLClassLoader.java" 445]
[clojure.lang.DynamicClassLoader findClass "DynamicClassLoader.java" 69]
[java.lang.ClassLoader loadClass "ClassLoader.java" 587]
[clojure.lang.DynamicClassLoader loadClass "DynamicClassLoader.java" 77]
[java.lang.ClassLoader loadClass "ClassLoader.java" 520]
[java.lang.Class forName0 "Class.java" -2]
[java.lang.Class forName "Class.java" 467]
[clojure.lang.RT classForName "RT.java" 2209]
[clojure.lang.RT classForName "RT.java" 2218]
[clojure.lang.Compiler resolveIn "Compiler.java" 7412]
[clojure.lang.Compiler resolve "Compiler.java" 7375]
[clojure.lang.Compiler analyzeSymbol "Compiler.java" 7336]
[clojure.lang.Compiler analyze "Compiler.java" 6785]
[clojure.lang.Compiler analyze "Compiler.java" 6762]
[clojure.lang.Compiler$MapExpr parse "Compiler.java" 3116]
[clojure.lang.Compiler analyze "Compiler.java" 6814]
[clojure.lang.Compiler analyze "Compiler.java" 6762]
[clojure.lang.Compiler$InvokeExpr parse "Compiler.java" 3900]
[clojure.lang.Compiler analyzeSeq "Compiler.java" 7126]
[clojure.lang.Compiler analyze "Compiler.java" 6806]
[clojure.lang.Compiler analyze "Compiler.java" 6762]
[clojure.lang.Compiler$BodyExpr$Parser parse "Compiler.java" 6137]
[clojure.lang.Compiler$FnMethod parse "Compiler.java" 5479]
[clojure.lang.Compiler$FnExpr parse "Compiler.java" 4041]
[clojure.lang.Compiler analyzeSeq "Compiler.java" 7122]
[clojure.lang.Compiler analyze "Compiler.java" 6806]
[clojure.lang.Compiler eval "Compiler.java" 7191]
[clojure.lang.Compiler eval "Compiler.java" 7149]
[clojure.core$eval invokeStatic "core.clj" 3215]
[clojure.core$eval invoke "core.clj" 3211]
[nrepl.middleware.interruptible_eval$evaluate$fn__968$fn__969 invoke "interruptible_eval.clj" 87]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[nrepl.middleware.interruptible_eval$evaluate$fn__968 invoke "interruptible_eval.clj" 87]
[clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
[clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
[clojure.main$repl$fn__9215 invoke "main.clj" 458]
[clojure.main$repl invokeStatic "main.clj" 458]
[clojure.main$repl doInvoke "main.clj" 368]
[clojure.lang.RestFn invoke "RestFn.java" 1523]
[nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
[nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
[nrepl.middleware.interruptible_eval$interruptible_eval$fn__999$fn__1003
invoke
"interruptible_eval.clj"
152]
[clojure.lang.AFn run "AFn.java" 22]
[nrepl.middleware.session$session_exec$main_loop__1067$fn__1071 invoke "session.clj" 202]
[nrepl.middleware.session$session_exec$main_loop__1067 invoke "session.clj" 201]
[clojure.lang.AFn run "AFn.java" 22]
[java.lang.Thread run "Thread.java" 833]]}#2023-05-1512:16roklenarciclet me make a gist#2023-05-1512:17roklenarcichttps://gist.github.com/RokLenarcic/ee3f0f16736ab6cc9c7ccb420607e46a#2023-05-1512:17delaguardodo you have somewhere in you class path such symbol: roklenarcic.edavki.currency?#2023-05-1512:17roklenarcicyes it’s the current namespace#2023-05-1512:17roklenarcic*ns*
=> #object[clojure.lang.Namespace 0x61f7236e "roklenarcic.edavki.currency"]
#2023-05-1512:19delaguardocan you re-evaluate it?#2023-05-1512:19roklenarcicLoading src/roklenarcic/edavki/currency.clj... done
(mi/collect!)
Syntax error (ClassNotFoundException) compiling at (/private/var/folders/fm/5mzhclpd7mj0tzjjq796ftc00000gn/T/form-init7469228970531054215.clj:1:1).
roklenarcic.edavki.currency
#2023-05-1512:19roklenarcicsame thing#2023-05-1512:20roklenarcicsource is just:
(ns roklenarcic.edavki.currency)
#2023-05-1512:25roklenarcicI’ll debug it#2023-05-1512:31roklenarcicI see… so the problem is with the fact that mi/collect! is a macro, so when you call it in the REPL it does weird things because the expansion of the macro requires running compiler and apparently it’s ran in some sort of limited environment#2023-05-1512:31delaguardothe source should be something like this:
(ns roklenarcic.edavki.currency
(:require [malli.instrument :as mi]))
(def small-int [:int {:max 6}])
(defn minus
"a normal clojure function, no dependencies to malli"
{:malli/schema [:=> [:cat :int] small-int]}
[x]
(dec x))
(mi/collect!)
#2023-05-1512:31roklenarcicso this works:
(mi/collect! {:ns 'roklenarcic.edavki.currency})#2023-05-1512:32roklenarcic(ns roklenarcic.edavki.currency
(:require [malli.instrument :as mi]))
(mi/collect!)
this doesn’t though#2023-05-1512:33delaguardoabove ^ works for me#2023-05-1512:34roklenarcicAh I wasn’t running 0.11.0#2023-05-1512:35roklenarcicworks after upgrading from 0.10.1#2023-05-1512:35roklenarcicok, but I need to have collect in each namespace, right?#2023-05-1512:36delaguardoyes, maybe there are some libs that can collect and instrument without explicit call, but I don't know them#2023-05-1512:38delaguardohave you read this - https://github.com/metosin/malli/blob/master/docs/function-schemas.md#tldr ?#2023-05-1512:51pithylessThere was this library before malli had its own instrumentation tooling - https://github.com/teknql/aave#2023-05-1514:03Craig BrozefskyIs there a library out there for generating markdown, or html or other "human readable" documentation of schemas. I tend to use schemas to define domain models (entities, value objects etc...) and having reference documentation, or even bringing up nice human readable tables or other formatted descriptions is very helpful for users who are not experirenced in deciphering edn on sight 8^)#2023-05-1518:32escherizeGlad you asked. We build some api docs using schemas, using this namespace: malli.experimental.describe#2023-05-1518:32escherizehttps://github.com/metosin/malli#description#2023-05-1701:07steveb8nQ: I’m using function schemas in jvm and node. I get excellent errors on the jvm but not great on node/cljs. by not great, I mean there’s no indication of the source or the error in the ex-info. Is this known or am I doing something wrong?#2023-05-1701:09steveb8nin the node server I can use binary search to find the source and then using m/explain to find the problem but this is slow. any suggestions for a faster way welcome#2023-05-1717:26sergey.shvetsI probably missing something obvious, but how do I validate that vector is a valid malli schema? I can run m/schema schema-def and it will throw, but I can't figure out if I can explain what's going on? E.g. "invalid predicate".#2023-05-1906:35Shuai Linhello channel 👋 is there a way to specify a map that
1. must have a key :foo
2. and must have one (and only one) of :bar or :baz ?
this shall work but looks awkward
[:and
[:map
[:foo :string]]
[:or
[:map [:bar :string]]
[:map [:baz :string]]]]#2023-05-1915:03tomcHi All,
Is there some shorthand for when an optional key is also allowed to be nil? For my purposes, if a key is absent that is the same as if it is present with the value nil . Right now I'm doing [:map [:k {:optional true} [:maybe string?]]] , but perhaps there is a better way?#2023-05-1921:06p-himikWould also love to have something like that. But at the very least, since it's all data, you can easily create your own wrapper function that produces that vector.#2023-05-1921:25escherizeI would use [:map [:k {:optional true} [:maybe string?]]].#2023-05-1921:26p-himikIt's fine for a one-off (or "few-off"), but doesn't feel great when it's repeated tens of times.#2023-05-1921:28escherizeWe have it in a few places. You can of course make code that returns [:k {:optional true} [:maybe string?]], but the current design is nice because optional is a property of the key, and maybe is a property of the value#2023-05-1921:36tomcThanks for the replies, all makes sense.#2023-05-1921:36escherizeUnrelated but in janet you cannot have nil as a value in a map for this reason!#2023-05-1921:37escherizeThis way to “dissoc” a key, you can “assoc” the key with a nil value. 🎉#2023-05-1921:38escherize(they call assoc put) https://janetdocs.com/put#2023-05-1921:38escherize(put @{:a 4 :b 5} :b nil)
# => @{:a 4}#2023-05-1921:46p-himikNot sure what you mean by "this reason", but it's rather practical to be able to distinguish between "there's no key" and "there's no value for this key". At the very least for easy communication with other entities that support nil-like things, e.g. relational databases.#2023-05-1921:47escherize“this reason” => it is often not a useful distinction (:a {}) vs (:a {:a nil}).#2023-05-1921:48escherizeI meant it as a response to:
> For my purposes, if a key is absent that is the same as if it is present with the value nil .#2023-05-1921:04James VickersHi all! I was trying out the new https://github.com/metosin/malli#malliexperimentaltime feature and had a couple questions about using it for JSON schema 🧵 Thanks much for an awesome library!#2023-05-1921:04James Vickers1. Is it intentional that https://github.com/metosin/malli/blob/master/src/malli/experimental/time/json_schema.cljc doesn’t have implementations for some of the specs like :time/instant? It seems like I’d need to implement malli.json-schema/accept for the missing specs to make them work properly for JSON schema.
2. I’m a bit surprised at the generated JSON schema for the new time specs; it uses references and is different than the corresponding predicate functions. I was hoping the new time keywords would generate a JSON schema similar to or the same as the corresponding predicate functions. I think this difference might be because of https://github.com/metosin/malli/blob/1a735fe5550b7dccf0832433f2f6d68ce9355edc/src/malli/core.cljc#L172 for these? Example:
(require '[malli.json-schema :as json-schema])
(json-schema/transform [:map [:t inst?]])
=> {:type "object", :properties {:t {:type "string", :format "date-time"}}, :required [:t]}
(json-schema/transform [:map [:t :time/instant]])
=> {:type "object", :properties {:t {:$ref "#/definitions/time~1instant"}}, :required [:t], :definitions #:time{:instant {}}}#2023-05-1921:40Ben SlessI'm partially to blame for your troubles, because I authored this implementation - I only implemented what's in the json schema spec, so maybe you should use a offset-date-time instead of an instant?#2023-05-2214:47James VickersThat wouldn’t have the same JSON representation or clojure/java type. We use java.time.Instant with ISO-8601 in JSON such as "2023-05-22T14:45:26.345322Z"#2023-05-2215:12Ben SlessIsn't the Z representing zone offset?#2023-05-2215:13James VickersI think so, but with no offset since Instant is always UTC time zone? Using OffsetDateTime would be a different type than Instant which adds some inconvenience.#2023-05-2215:30Ben SlessAFAIK instance has no zone, not a default zone,evident by it having no method to get an offset or zone from it, only adding them to it, getting a different time.
To make things more annoying, there is no equivalent in Json schema, only offset date time.#2023-05-2215:31Ben Slesshttps://docs.oracle.com/javase/8/docs/api/java/time/Instant.html#2023-05-2215:35James VickersUsing inst? in a malli schema gives {:type "string", :format "date-time"} . I’m thinking :time/instant and inst? should preferably produce the same JSON schema.#2023-05-2215:39Ben Slesshttps://datatracker.ietf.org/doc/html/rfc3339#section-5.6
full-time = partial-time time-offset
date-time = full-date "T" full-time#2023-05-2215:40Ben SlessIt's wrong#2023-05-2218:07James VickersI’m confused what’s wrong specifically, can you clarify? The way inst? is serialized is wrong? The JSON schema that malli produces for inst? makes sense to me, since it’s an ISO-8601 string with an optional offset or just Z when there is no offset (e.g. UTC). The date-time format also matches how java.time.Instant is serialized to JSON by clojure and java libraries, as ISO-8601 strings. It also still seems to me like there’s a pain point and confusion if :time/instant and inst? generate different JSON schema representations even though they both are java.time.Instant and have the same JSON string representation.#2023-05-2004:28aaron51What’s the recommended way to do server-side rendering of error messages on form fields, with reitit+malli?
If I use malli to validate reitit params, then invalid requests just get a server-side error — I want to re-render the form with user-visible humanized error messages on each field.
(I don’t want to add the error messages client-side)#2023-05-2005:46hifumi123Use malli.error/humanize for this and ensure your form has unique names#2023-05-2005:49hifumi123(def form-data-schema
[:map
[:foo :string]
[:bar [:int {:min 1 :max 10}]]
[:baz [:map
[:quux :boolean]]]])
(->> {:foo "" :bar 0 :baz {:quux nil}}
(m/explain form-data-schema)
(me/humanize))
;; => {:bar ["should be between 1 and 10"]
;; :baz {:quux ["should be a boolean"]}}#2023-05-2005:50hifumi123you can also customize the error messages with :error/message or :error/fn#2023-05-2015:04SamHi,
any suggestions on how to clean up my code?
It's a function for an API. It takes a function, the function input/output schemas, and function input args.
It validates the input args using malli and the input schema. If the input is of the wrong format it returns a bad-request with info from malli.
It then runs the function and verifies the output schema using malli, then returning either the function output or the malli error message.
The schemas arg is a map of input/output schemas, for example:
{:input
[:map
[:url :string]]
:output
[:string]}
Here's the somewhat verbose function
(defn handler-wrapper [function schemas args]
(if-let [bad-input (me/humanize (explain-data (:input schemas) args))]
(bad-request {:error "Invalid input schema"
:information bad-input})
(let [output (function args)
bad-output (me/humanize (explain-data (:output schemas) output))]
(if bad-output
(bad-request {:error "Internal error"
:information bad-output})
(response output)))))#2023-05-2015:44p-himikLGTM, I don't see anything worth changing.#2023-05-2015:45SamCool, thank you!#2023-05-2120:14Karol WójcikHey,
I’m using function schemas and I’m wondering if it’s possible to include function name and schema name in the instrumentation error?#2023-05-2120:50p-himikWhen you provide a custom report function to instrument!, the function name will be included.
The schema is included by default.
Adding to the power example at https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defn-instrumentation:
(mi/instrument! {:report m/-fail!})
..instrumented #'dev/power
=> nil
(power 6)
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
:malli.core/invalid-output
(ex-data *e)
=>
{:type :malli.core/invalid-output,
:message :malli.core/invalid-output,
:data {:output [:int {:max 6}], :value 36, :args [6], :schema [:=> [:cat :int] [:int {:max 6}]], :fn-name dev/power}}#2023-05-2210:08Karol WójcikCan it also return the schema name?#2023-05-2210:11p-himikWhat is "schema name"?#2023-05-2210:16p-himikIf you mean that in
(def my-schema [:map [:a int?]])
the my-schema symbol names the schema, then no.
Schemas are just data (unless you have already called m/schema on it), so the don't know the var that was used to store them.
You can, however, attach arbitrary data to a schema and it will be reported as well:
(m/explain [:map {::schema-name "An int under :a"} [:a int?]] {:a ""})
=>
{:schema [:map #:dev{:schema-name "An int under :a"} [:a int?]],
:value {:a ""},
:errors ({:path [:a], :in [:a], :schema int?, :value ""})}#2023-05-2210:17p-himikIt's mentioned in the documentation here https://github.com/metosin/malli#validation in the code block under "Schemas can have properties".#2023-05-2121:43escherizeI did a little experiment to make a little pattern matching macro that works with malli multi-schemas. source in 🧵 Any thoughts?#2023-05-2121:44escherize(defmacro match-maker
"Return a function that takes 2 things:
a malli multi schema M,
pairs of multi schema dispatch values -> functions
And returns a function, that when given a value that adheres to the
multischema M, calls the dispatch function and calls the right value on it.
"
[schema & efp]
`(let [_# (#'clojure.core/assert-args (even? (count '~efp)) "an even number of forms in evp vector")
enum->fn-pairs# (mapv vec (partition 2 '~efp))
children# (m/children ~schema)
expected-branches# (mapv first children#)
found-branches# (mapv first enum->fn-pairs#)
dispatch-val->f# (into {} (eval enum->fn-pairs#))
dispatch-fn# (-> ~schema m/properties :dispatch)]
(when (not= expected-branches# found-branches#)
(throw (ex-info (str "Error: mismatched branches in match-maker."
"\nExpected: " (pr-str expected-branches#)
"\nReceived: " (pr-str found-branches#))
{:exptected expected-branches#
:found found-branches#})))
(fn [value#]
(let [branch# (dispatch-fn# value#)
branch-fn# (get dispatch-val->f# branch#)]
(branch-fn# value#)))))#2023-05-2121:45escherize;; Given a multi schema:
(def PetSchema
[:multi {:dispatch :type}
[:dog [:map [:good-dog? :boolean]]]
[:cat [:map [:temperment [:enum :nice :mean]]]]])
;; make a function that can dispatch on the keys
(def pet-noise
(match-maker PetSchema
:dog (fn [{:keys [good-dog?]}] (if good-dog? "good dog." "bad dog."))
:cat (fn [{:keys [temperment]}] (str "kitty is " (name temperment)))))
;; call the function
(mapv pet-noise
[{:type :dog :good-dog? true} {:type :dog :good-dog? false}
{:type :cat :temperment :nice} {:type :cat :temperment :mean}])
;; => ["good dog." "bad dog." "kitty is nice" "kitty is mean"]#2023-05-2121:47escherizeThe nice thing about it is when you update the schema:#2023-05-2121:48escherizeupdate the schema with a new branch
(def PetSchema
[:multi {:dispatch :type}
[:dog [:map [:good-dog? :boolean]]]
[:cat [:map [:temperment [:enum :nice :mean]]]]
[:bird [:map]]])
And try to use match-maker on the updated schema, you’ll get an error because you didn’t handle :bird!
(try (def pet-noise
(match-maker PetSchema
:dog (fn [{:keys [good-dog?]}] (if good-dog? "good dog." "bad dog."))
:cat (fn [{:keys [temperment]}] (str "kitty is " (name temperment)))))
(catch Exception e [(ex-message e) (ex-data e)]))
;; => ["Missing Branches.
;; Expected: [:dog :cat :bird]
;; Received: [:dog :cat]"
;; {:exptected [:dog :cat :bird], :found [:dog :cat]}]#2023-05-2210:01hifumi123Is there a schema for maps whose keys are all namespace qualified to a specific namespace? I don’t want to type out the same namespace N times, so that disqualifies something like`[:and [:fn (fn [m] (every? (comp (partial = "ns") namespace) (keys m)))] [:map …]]`#2023-05-2210:08p-himikDon't think there's such a schema but you can write a schema transformer function that accepts [:map [:a int?] [:b string?]] along with :my-ns and spits out [:map [:my-ns/a int?] [:my-ns/b string?]].#2023-05-2210:10p-himik> I don’t want to type out the same namespace N times
You can also use namespace aliases to alleviate the issue a bit.
Another option would be to use malli.experimental.lite/schema:
(malli.experimental.lite/schema #:some-ns{:a int?, :b string?})
=> [:map [:some-ns/a int?] [:some-ns/b string?]]
#2023-05-2210:11hifumi123I'm leaning towards the transformer option; thanks!#2023-05-2307:36SamWhat is the syntax to assign an optional key a default value?
I thought something like this would work:
[:map
:optkey {:optional true} [:string {:default "Default value"}]
:regular-key :string]
Meaning: If optkey is provided use that value, if it is omitted then set its value to "Default Value"#2023-05-2307:43p-himikIt's a property of the key, so should be in the same map where you specified that it's optional.#2023-05-2307:44SamThank you!#2023-05-2307:45p-himikOh, hmm, or maybe I'm wrong now that I've double-checked the docs.#2023-05-2307:46p-himikThe docs use the syntax in your OP but also mention that
> Optional Keys are not added by default
But apparently it's possible to use ::mt/add-optional-keys to add them.#2023-05-2307:47p-himikYeah, the syntax doesn't matter - works both ways. But the default is added only with that option.#2023-05-2307:49SamI see, that works. Thank you!#2023-05-2315:02Noah BogartHow exactly does Malli instrument macros? I'm reading through the code and I don't see where it hooks in (outside of alter-var-root)#2023-05-2315:04escherizedid you look at how -strument works?#2023-05-2315:04escherizemalli.instrument/-strument! I mean#2023-05-2315:05escherizeIt ends up calling malli.core/-instrument to alter-var-root. Is that what you meant tho?#2023-05-2315:06Noah BogartI don't think I was specific enough. in Compiler.java, clojure calls out to clojure.spec.alpha/macroexpand-check during macro-expansion to throw exceptions if a macro is instrumented by a spec. malli isn't keyed into that, so I would expect that macro-expansion happens before malli's instrumentation can be applied#2023-05-2315:06Noah Bogartbut i think i'm missing a part of the interaction between macros and...#2023-05-2315:06Noah Bogartactually, i think i just got it lmao#2023-05-2315:07Noah Bogartthank you for rubber ducking#2023-05-2315:08escherizenow I’m curious: what did you figure out 🙃#2023-05-2315:09Noah Bogartto explain, macros are "just" functions that have the :macro true meta. because of that, they're called during macro-expansion in the compiler, not at run-time. when a macro's root var is altered to be a different function, that new function is what is called at macro-expansion time, and the underlying/original macro function is passed all of the given args if and only if validation is successful#2023-05-2315:11Noah Bogarteasy peasy#2023-05-2315:15escherizenicely said: and I guess that since malli instrumentation doesn’t use a macro, it just swaps out the function it’s instrumenting, that you dont need to worry about the macroexpand-check ? Or are you going to instrument a macro? 😱#2023-05-2315:17Noah Bogartthe call to clojure.spec.alpha/macroexpand-check only does something if there's a spec for that particular var and in my case there is not, so it's a no-op. i was thinking about instrumenting a macro, it's helpful to have that kind of feedback during development#2023-05-2315:09Noah Bogartto explain, macros are "just" functions that have the :macro true meta. because of that, they're called during macro-expansion in the compiler, not at run-time. when a macro's root var is altered to be a different function, that new function is what is called at macro-expansion time, and the underlying/original macro function is passed all of the given args if and only if validation is successful#2023-05-2320:38Sam RitchieI’m reading the docs but can’t find the answer to this (apologies if it is easy!)
Can I use malli to define a function in some way that generates a docstring?#2023-05-2320:39Sam Ritchie(defn point
"Constructs a point fragment.
- `:x`: x-coordinate of the point.
- `:y` y-coordinate of the point.
- `:color`: any valid [CSS
color](), or any keyword
from [[mafs.core/Theme]].
- `:opacity`: Double in the range [0.0, 0.1] inclusive.
- `:svg-circle-props`"
[pair-or-m]
...)
I’m writing something like this, and it would be great to have the docs generated from the types#2023-05-2320:55escherizeNot really, but there’s some prior art#2023-05-2320:56escherizehttps://github.com/metabase/metabase/blob/master/src/metabase/util/malli.cljc#L94-L101#2023-05-2320:56escherizeprobably worth contributing this back into malli#2023-05-2321:37steveb8nhttps://clojurians.slack.com/archives/CLDK6MFMK/p1680658917653729?thread_ts=1680658917.653729&cid=CLDK6MFMK#2023-05-2405:41CarloA couple of questions in 🧵 on this simple example:
(def ex-spec-1
[:map [:foo :keyword]])
(def ex-spec-2
[:* ex-spec-1])
(m/explain ex-spec-2 [{:foo 3} {:foo :a}])
which returns:
{:schema [:* [:map [:foo :keyword]]],
:value [{:foo 3} {:foo :a}],
:errors
({:path [0 :foo], :in [0 :foo], :schema :keyword, :value 3}
{:path [],
:in [0],
:schema [:* [:map [:foo :keyword]]],
:value {:foo 3},
:type :malli.core/input-remaining})}#2023-05-2405:44Carlo1. you can see that in the schema key in the return value the definition of the schema is expanded out. This is fine in this case, but when you have a lot of nested schema definitions, that becomes a couple pages of unreadable mess. Is there a way I can stop displaying at the first level, in this case [:* ex-spec-1]?
2. is there ever a difference between defining specs how I did it above wrapping the vector specification with malli.core/schema, like (m/schema [:map [:foo :keyword]])? What's the benefit of having both versions?
3. what's the :malli.core/input-remaining error about? Just signaling that there might be other errors?#2023-05-2406:26opqdonut3: input-remaining means that the :* sequence expression didn't manage to consume the whole sequence, i.e. there was extra stuff left over#2023-05-2406:27opqdonut2: if you use malli functions, they'll call m/schema for you, which will result in the schema getting recompiled. That can be a performance problem if the schema is big or used really often. Other than that, it doesn't matter.#2023-05-2406:27opqdonut1: that's a good question. I sometimes end up dissocing :schema from my errors because it's not helping. Using a custom registry might help, but I haven't tried it.#2023-05-2406:29opqdonutdid that help?#2023-05-2406:30CarloYes, thank you very much @US1LTFF6D!#2023-05-2406:30CarloI'll share here if I end up with a solution for 1, but in the meantime I'll probably dissoc schema too!#2023-05-2414:17Noah Bogartthere's also me/humanize#2023-05-2414:17Noah Bogarthttps://cljdoc.org/d/metosin/malli/0.11.0/doc/readme#humanized-error-messages#2023-05-2414:19CarloYes, thank @UEENNMX0T, I'm using this from (dev/start! {:report (pretty/reporter)}) so the errors appear automatically in my repl when I run instrumented code. I usually find humanized messages a bit of a hit-or-miss. Many times it's just "Invalid type" 😂#2023-05-2414:24Noah Bogartyeah i wish the humanized errors were better about telling you the difference#2023-05-2414:25Noah Bogartbut they're better than vomiting 10 mb of edn into my console lol#2023-05-2414:48escherizeA trick I use for humanization is to print out the received value. You do that by calling humanize with a :wrap function like so:
(me/humanize
(m/explain input args)
{:wrap
(fn [{:keys [value message]}]
(str message ", received : " (pr-str value)))})#2023-05-2414:49escherizeit doesn’t always help, but it can help figure out which part of your value was incorrect#2023-05-2414:49Carlowhat's tru?#2023-05-2414:50escherizetypo 😉#2023-05-2414:50escherizehttps://github.com/metabase/metabase/blob/bcm-typed-collections/src/metabase/util/malli.cljc#L44-L68#2023-05-2414:50Noah Bogartoh that's nice#2023-05-2414:50escherizeit’s for translation at metabase#2023-05-2516:10ingesolIs it possible in malli to instrument a function on the fly? My use case is wanting to ensure symmetric re-frame db handlers that always receive and return the db. I would like to instrument programmatically in my reg-event function. Example:
(defn my-updater-event [db k v]
(update db k v))
(instrument-this-fn my-updater-event [:=> [:cat :map [:* :any]] :map])
#2023-05-2516:16p-himikIt's possible in CLJ with alter-var-root - and that's how built-in Malli instrumentation works.
In CLJS, it might be possible with set!: https://clojurians.slack.com/archives/C03S1L9DN/p1604308689481100
But no clue whether it'll survive advanced optimizations and, even if it does now, whether it's something one can rely on.
I would probably write a macro that works like defn but also accepts a schema and produces an already instrumented function.#2023-05-2516:25ingesolyup, I did the macro experiment earlier and it’s pretty nice. Probably makes more sense.#2023-05-2516:37escherizeThere is already such a function, in https://github.com/metosin/malli/blob/master/src/malli/experimental.cljc#L71#2023-05-2516:38escherizeI havn’t tried but it looks like it’ll work in cljs. You can add :malli/always to the function’s metadata to make malli.. always check the input and outputs#2023-05-2607:43ingesolThanks, know about that one. The variant I want is a version of this macro that always enforces a certain schema, so the same but actually simpler.#2023-05-2900:13steveb8nQ: @ikitommi can you say how you expect mx/defn to progress? We’re liking it and starting to use it. It would help to know if it is likely to graduate out of mx.#2023-05-2905:17ikitommiUsing that too, I think the question should be in malli.core or a separate ns. But, it’s stable. There will be mx/def too (https://github.com/metosin/malli/pull/776)#2023-05-2905:18ikitommiI think it will stay out of core, but you can use that.#2023-05-2907:11steveb8nok. great. we love it 🙂 thanks#2023-05-2907:12steveb8nQ: is anyone else using Malli in a node.js runtime? specifically we are using function schemas in node (shadow) and are not getting clear error messages (compared to jvm) when inputs or outputs fail validation#2023-05-2907:13steveb8nI see these instructions https://github.com/metosin/malli/blob/master/docs/clojurescript-function-instrumentation.md but I wonder if anyone has figured out how to make this work for a non-browser runtime i.e. node server#2023-06-2202:43steveb8n@ikitommi do you know if anyone has made fn schemas and dev friendly logs work on node.js? seems like it’s good in the browser but validation errors are quite opaque on node.#2023-06-0712:24igrishaevI'm not getting how does malli handle transformation. I'd like to have a schema email-address that:
• ensures it's a string
• trims and lower-cases it
• ensures there is @ in a string
Here is what I've got so far:
(def email-address
[:fn
{:error/message "should be a proper email address"}
(fn [x]
(when (string? x)
(let [email
(-> x
(str/trim)
(str/lower-case))]
(when (str/includes? email "@")
email))))])
Suddenly, it doesn't work with m/decode: I'm still getting a weird string like sdfsdfsf\n\n\n whereas I expected an error#2023-06-0712:26igrishaevMore details on that: in our project we have a bottleneck like this:
(m/decode schema item item-transformer)
where item-transformer is
(def item-transformer
(mt/transformer
mt/strip-extra-keys-transformer
(mt/default-value-transformer {::mt/add-optional-keys true})
mt/string-transformer))
Does it mean I must introduce a new transformer?#2023-06-0712:34igrishaevAlso, I expected m/decode would return a value from a function specified in :fn, but instead, it returns the origin (raw) value#2023-06-0713:43igrishaevok here is a working version
(def email-address
[:string
{:re #"(.+?)@(.+)"
:error/message "Should be a non-empty string containing @"
:decode/string (comp str/lower-case str/trim)}])#2023-06-0714:18escherizedecode doesn’t do validation. maybe you want coerce#2023-06-0815:45sparkofreasonWhen using providers, is there a way to infer that a string valued property is an enum?#2023-06-0816:02p-himikJudging by the source code, I don't see any way other than
(mp/provide
[(mp/-hinted "a" :enum)
(mp/-hinted "b" :enum)])
=> [:enum "a" "b"]
#2023-06-0816:03p-himikAh, hinting just the first value works as well:
(mp/provide
[(mp/-hinted "a" :enum)
"b" "c" "d"])
=> [:enum "a" "b" "c" "d"]
#2023-06-0816:06sparkofreasonThanks. My case is more like digging in to a map. So given something like:
[{:data {:stage "foo"}}
{:data {:stage "bar"}}
...]
I'd like to infer [:map [:data [:map [:stage [:enum "foo" "bar"]]]]].#2023-06-0816:17sparkofreasonSo maybe the correct question is how to hint a specific property.#2023-06-0816:30p-himik(mp/provide
[{:data {:stage (mp/-hinted "foo" :enum)}}
{:data {:stage "bar"}}])
=> [:map [:data [:map [:stage [:enum "foo" "bar"]]]]]#2023-06-0816:32sparkofreasonOh, now that is cool. Thanks!#2023-06-0819:04roklenarcicWhen my [:fn pred] pred function throws some exception I only get an empty message#2023-06-0819:04roklenarcichow do I get the actual exception#2023-06-0819:21p-himikNo built-in way.
But if that exception has :type in its ex-data, that will become a part of the reported error.#2023-06-0820:08roklenarcicUnfortunately it doesn’t, the exception was :
Execution error (IllegalArgumentException) at tick.protocols/eval11136$fn$G (protocols.cljc:88).
No implementation of method: :date of protocol: #'tick.protocols/IExtraction found for class: nil
so what I got was almost like a silent failure#2023-06-1318:47mafcocincoWhat is the best practice for getting malli to include the value that is violating a schema in the context of a :tuple or :sequential? For example:
(-> (m/explain [:tuple :int :int] ["foo" 1]) (m.e/humanize))
;; => [["should be an integer"]]
#2023-06-1318:47mafcocincoWould be helpful if the should be an integer error included "foo" as the value that is violating the schema.#2023-06-1318:48mafcocincoI assume this is probably done with some kind of custom resolver within the humanize function?#2023-06-1318:49ikitommithere is a :wrap option, see https://cljdoc.org/d/metosin/malli/0.11.0/doc/tips#getting-error-values-into-humanized-result#2023-06-1319:04mafcocincosweet. That worked great! Thank you for your help @ikitommi#2023-06-1413:47Noah Bogartjust for fun, i've been experimenting with -vmap and I can't seem to get personal benchmarks on my own computer to run any faster than with normal mapv. is this a quirk of my machine or maybe an increase in clojure's speed since october 2021?#2023-06-1415:06Ben SlessCould be a function of jvm version, input size, clojure version,CPU architecture
Need a comprehensive matrix#2023-06-1415:08Noah Bogartyeah that makes sense. i won't dive into all that then, i'll just chalk it up to a quirk of my set up.#2023-06-1415:10Ben SlessIt's worth revisiting#2023-06-1514:55ikitommi@UEENNMX0T do you have flamegraphs of -vmap vs mapv?#2023-06-1514:56Noah Bogarti don't have them still, but when i looked, i did see a reduction of stack frames: mapv -> reduce -> persistentvector.reduce etc, compared to vmap's single frame#2023-06-1515:05Noah Bogarti did this on my 2019 macbook pro with clojure 1.11 and java 20, which i suspect is doing a lot of work here. i didn't try it on earlier versions of either#2023-06-1515:08ikitommiI have not run the benchmarks on >java17. Should re-visit the tests with latest Java. There are some of perf tests in malli repo.#2023-06-1515:09Noah Bogarti'll give them a try, see how they do#2023-06-1413:47Noah Bogartor is it more that it's roughly the same speed but provides a consistent api for all "map"-like needs in malli and so you rely on it over everything else#2023-06-1515:00fabraoHello all, if you have a nested schema, are you able to access the nested keys from the parent?#2023-06-1515:07ikitommiyes, there are utilities for this in malli.util, e.g. get, get-in , …#2023-06-1517:45fabraothank you#2023-06-1517:58fabraoSo, how the :fn of [:and [:....][:fn ...][:fn ...][:fn ...]] runs? In sequence?#2023-06-1518:19escherizeWell, let’s see:
(m/validate [:and keyword?
[:fn (fn [_] (println 1) true)]
[:fn (fn [_] (println 2) true)]
[:fn (fn [_] (println 3) true)]
[:fn (fn [_] (println 4) true)]]
:k)
;; =stout=>
;; 1
;; 2
;; 3
;; 4
;; => true#2023-06-1603:14fabraoSorry, I mean serialized? Is it can be run in parallel / thread?#2023-06-1721:06souenzzoyes, serialized
why do you think it would be better to run in parallel?#2023-06-2015:03fabraowell, performance?#2023-06-2015:13souenzzoDoes Parallel have more performance than serialized?
Imagine the situation:
You have a schema with 10 and elements
Your app receives 10 requests, that your http server will process inside a thread pool of 10.
Each request calls the validation.
If it was parallelized, you will have 100 parallel threads at this point. It may cause performance issues in your 8 cores machine.
Now imagine that inside the and, it usually fails in the 2-3 element.
All left 7-8 functions will be evaluated, as they are all spawned at the same time#2023-06-2015:15fabraohummm, that's a good point, thank you for your scenario vision#2023-06-2015:20souenzzoMaybe in your scenario it makes sense to have it parallelized, but it is not a generic "better" thing
Do benchmarks, have metrics, before move into this approach.
If that is the case, I think that a good start point is copy (defn -and-schema ... from malli.core into your code and define your :custom.parallel/and schema.#2023-06-2015:22souenzzoMy problematic scenario can be "solved" by using a thread pool for validation, then you will have a constant number of threads (http thread pool size + validation thread pool size).#2023-06-2015:23fabrao"My problematic scenario can be "solved" by using a thread pool for validation" that I was thinking to have#2023-06-2015:26souenzzoIt is doable inside :your-custom/and#2023-06-1920:01Olav FosseHey, is there a way to infer schemas which detect tagged unions? Would be super useful when looking at big nested values.#2023-06-1921:43hifumi123assuming :type is required, I would make a multi-schema dispatching on that value#2023-06-2017:12ikitommiThere is no inferring of :multi atm. Please write an issue of this, I think it would be valuable. I think this should not be enabled by default - easy to go wrong with this.#2023-06-2017:14ikitommisimilar case with :map-of - finding a default threshold where it gives mostly correct values proved to be hard, so disabled by default (I recall)#2023-06-2015:09fabraoHello all, is coercion can be applied to explain process?#2023-06-2017:14ikitommiNot sure what you mean by this.#2023-06-2114:01fabraowell, I have a ring process with route that have post with coercions for checking requests and responses. And it configure :strip-extra-keys as false , that results into some problems that I don't have in explain process. What can I do to have the same behavior into explain? Thank you#2023-06-2114:13Nikolas PafitisHello people, I'm trying to infer a schema from a map using malli.provider
like so:
(mp/provide [{:assumed-infiltration 50,
:building-air-leakage 6,
:building-air-leakage-units "ACH",
:building-type "single-family detached",
:conditioned-floor-area 2600,
:cooling-systems [{:efficiency 13,
:efficiency-units "SEER",
:load 1,
:name "CentralAirConditioner1",
:system-type "central air conditioner"}],
:heating-systems [{:efficiency 0.8,
:efficiency-units "AFUE",
:fuel "natural gas",
:load 1,
:name "Furnace1",
:system-type nil}],
:roadmap #uuid "c4e39533-e4d1-4a31-87a1-53f5c73100e5",
:roof-area 1453,
:thermostat {:cooling-setpoint 78,
:heating-setpoint 68,
:name "HVACControl1",
:system-type "manual thermostat"},
:water-heaters [{:energy-factor 0.59,
:fuel-type "natural gas",
:load 1,
:location "living space",
:name "WaterHeater1",
:system-type "storage water heater",
:tank-volume 40,
:temperature 125}]}
{:assumed-infiltration 50,
:building-air-leakage 6,
:building-air-leakage-units "ACH",
:building-type "single-family detached",
:conditioned-floor-area 2600,
:cooling-systems [{:efficiency 13,
:efficiency-units "SEER",
:load 1,
:name "CentralAirConditioner1",
:system-type "central air conditioner"}],
:heating-systems [{:efficiency 0.8,
:efficiency-units "AFUE",
:fuel "natural gas",
:load 1,
:name "Furnace1",
:system-type nil}],
:roadmap #uuid "c4e39533-e4d1-4a31-87a1-53f5c73100e5",
:roof-area 1453,
:thermostat {:cooling-setpoint 78,
:heating-setpoint 68,
:name "HVACControl1",
:system-type "manual thermostat"},
:water-heaters [{:energy-factor 0.59,
:fuel-type "natural gas",
:load 1,
:location "living space",
:name "WaterHeater1",
:system-type "storage water heater",
:tank-volume 40,
:temperature 125}]}])
but I get this stack trace back:
nrepl.middleware.interruptible-eval/evaluate/fn interruptible_eval.clj: 87
...
clojure.core/with-bindings* core.clj: 1990 (repeats 2 times)
clojure.core/apply core.clj: 667
...
nrepl.middleware.interruptible-eval/evaluate/fn/fn interruptible_eval.clj: 87
clojure.core/eval core.clj: 3221
...
user/eval82631 REPL Input
malli.provider/provide provider.cljc: 106
malli.provider/provide provider.cljc: 107
malli.provider/provider provider.cljc: 100
malli.provider/-inferrer provider.cljc: 16
clojure.core/mapv core.clj: 6984
clojure.core/reduce core.clj: 6900
clojure.core.protocols/fn/G protocols.clj: 13
clojure.core.protocols/fn protocols.clj: 75
clojure.core.protocols/seq-reduce protocols.clj: 31
clojure.core.protocols/fn/G protocols.clj: 19
clojure.core.protocols/fn protocols.clj: 168
clojure.core/mapv/fn core.clj: 6993
clojure.core/juxt/fn core.clj: 2611
malli.core/validator core.cljc: 2043
malli.core/validator core.cljc: 2045
malli.core/-cached core.cljc: 283
malli.core/eval22682/fn/G core.cljc: 28 (repeats 2 times)
malli.util/-util-schema/reify/reify/-validator util.cljc: 375
malli.core/eval22682/fn/G core.cljc: 28
clojure.core/-cache-protocol-fn core_deftype.clj: 584
java.lang.IllegalArgumentException: No implementation of method: :-validator of protocol: #'malli.core/Schema found for class: nil
I'm using mutable-registry if that's important.#2023-06-2114:17Nikolas PafitisEven a simple example like so
(mp/provide [{:assumed-infiltration 50}])
throws the same exception. I'm using version 0.11.0 had the same error with 0.8.4#2023-06-2222:01p-himikThat very line works for me just fine on 0.11.0.
Perhaps you managed to reload some code from Malli that has already been loaded. Or you have some malli namespace on classpath that's somehow not compatible with the current version.#2023-06-2221:58didibusIn a map spec, can you refer to other specs unqualified? The doc only mentions qualified keys: https://github.com/metosin/malli#qualified-keys-in-a-map
And just in general, how would I refer to another spec in Malli? Say I want to create an age spec which is a range from 1 to 120? For example? Or a email type for email and I want to then use that in lists and maps and all that?#2023-06-2222:01didibusLike say given this example:
(def Address
[:map
[:id string?]
[:tags [:set keyword?]]
[:address
[:map
[:street string?]
[:city string?]
[:zip int?]
[:lonlat [:tuple double? double?]]]]])
I want to make city an enum of say 500 cities, and I want to define that separately and refer it here? And I might refer it in other places, but I don't want the key to become ::city
The wiki seems to define everything inline.#2023-06-2300:12Michael Gardnerput it in a var or binding, just like any other data
(def City
[:enum "Ahmedabad" "Alexandria" ...])
(def Address
[:map
[:address
[:city City]]])#2023-06-2300:14Michael Gardnerwe do this even for widely-used schemas. We just put the var in a common schemas namespace#2023-06-2300:19didibusI guess that's a simple way. I also found you can use mutable registry and set default registry. I won't what the pros and cons are as opposed to what you suggested#2023-06-2303:50Ben SlessFrom experience using def forms, use a registry and set it as default#2023-06-2303:58didibus@UK0810AQ2 Just making sure I'm not reading your sentence wrong, you're saying, from your experience, it's better to use a mutable registry and set it as default, than using def forms?#2023-06-2304:09Ben SlessIt doesn't have to be mutable (you also don't have to set it as default, you can use a schema schema), but yes, it's preferable to using def#2023-06-2306:43ikitommi> .. And I might refer it in other places, but I don't want the key to become ::city
>
You can do:
1. [:map ::city] - qualified key + registry
2. [:map [:city ::city]] - any key + registry
3. [:map [:city City]] - any key + Var
At library code, you can't rely on the existence of the global registry, so 3 (or Schemas with explicit immutable registry) should be used. Used them all, leaning bit towards the global registry.#2023-06-2306:44ikitommiand, documentation improvement most welcome!#2023-06-2321:32Michael Gardner@UK0810AQ2 why do you find it preferable?#2023-06-2310:50didibusAlso curious if there's been any thought around something similar to Spec2 schema/select for Malli? Any experiment on that front?#2023-06-2312:29valtterihttps://github.com/metosin/malli/issues/95#2023-06-2312:45eval2020It comes up once in a while: https://clojurians.slack.com/archives/CLDK6MFMK/p1678389151150529#2023-06-2313:03bortexzAlso: https://github.com/metosin/malli/issues/724#2023-06-2313:27Craig BrozefskyI regularly use malli schema transform operators to do this -- including defining input and storage derivatives of schemas that identify my domain entities. I have also used the utils for making keys required or optional, to build derivative schermas for input and "compiled" forms for expressions in DSLs. Malli schema are data....#2023-06-2313:31Craig Brozefskysee malli.util(mu). mu/required-keys, mu/optional-keys, mu/merge, mu/select-keys, mu/closed-schema, mu/open-schema ....#2023-06-2313:34Craig BrozefskySo I can define a "canonical" schema for an entity, then use those to derive input (aka, POSTing) schema. Those would disallow some keys, like the computed id. Same with storage, which might have more "computed" fields like timestamps.#2023-06-2318:02hifumi123yup, I use malli.util fairly often too. mu/keys is especially useful in frontend applications when I'm submitting form data and want to submit only the actual keys ive validated for the form#2023-06-2318:07didibusCool, I'll take a look at these.
I've been using multi-spec in Spec1. But it's not as ergonomic, since you have to manually copy/paste all versions of your schema, and then you don't have a place where you can see the full union.#2023-06-2318:20didibusI guess there's not something exactly like select, that would make everything optional except for the selected keys which would be required. But I can probably do something myself that does this.#2023-06-2318:23hifumi123you should be able to use a mix of union, selected-keys, and required-keys to recreate the behavior#2023-06-2318:24hifumi123(mu/union (mu/optional-keys schema)
(mu/required-keys (mu/select-keys schema ks)))#2023-06-2318:24hifumi123i would guess this#2023-06-2318:36didibusOh, nice, ya if optional-keys has an arity that just optionals everything, that should work.#2023-09-2214:57eval2020I recently released https://github.com/eval/malli-select to allow for spec2-like selection of malli-schemas. Feedback welcome!#2023-09-2415:42ikitommiNice! I’ll add this to README and mention it in the malli 0.13.0 announcement 🥳#2023-06-2317:56fabraoHello all, is there possible to include some Vendor extension into the parameter of swagger schema result, the x-examples?
Like here:
... "paths" : {"/v1/show-informations" : { "post" : { "parameters": [{ ..., "x-examples": { ...
?#2023-06-2709:22ikitommidoes adding :swagger/x-examples work? e.g. [:int {:swagger/x-examples [1,2,3]}]#2023-06-2715:20fabraoNo, it should be as another parameter like:
"parameters": [
{
"in": "body",
"name": "Request Timelines Payload",
"description": "",
"required": true,
"schema": {...},
"x-examples" : {...
},
but if I use "swagger/x-examples": *{*"x-examples": *{ ...* into :parameters it's including outsite of in: body , like
"parameters": [
{
"in": "body",
"name": "Request Timelines Payload",
"description": "",
"required": true,
"schema": {}
},
[
"swagger/x-examples",
{
"application/json": {
"simple-example": {
"summary": "Simple Example",
"values": {
"someRequest": "REQUEST INFORMATION"
}
}
}
}
]
I want to know how to include it into in body, if possible#2023-06-2320:36mafcocincoI noticed in there is a start! and stop! function. It seems these functions, or at least start! would be useful for instrumenting all functions that were in scope outside of a development context. The name of namespace gave me pause WRT using start! in production code. Would likely not need to call stop! outside of a dev context but I prefer a single function to instrument all functions in a single stop rather then having to sprinkle mi/collect! calls throughout all namespaces that use function schemas.
So, 2 questions:
1. is there any gotchas/harm in using outside of a development context (re: production code)?
2. Is calling mi/collect! within each namespace the accepted way of instrumenting functions using defn metadata or is there a better/similar way to !?#2023-06-2709:12ikitommihi. You could use in prod, or just call the relevant malli.instrument functions in case you don’t need things like automatic Var watching, clj-kondo settings, pretty errors etc. e.g. the stack is:
1. malli.core/-instrument - pure functions
2. malli.instument/collect!|instrument! - side-effecting one-time functions
3. ! - side-effecting, adds developer things like pretty-printing#2023-06-2709:12ikitommisee https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defn-instrumentation#2023-06-2813:32mafcocincoYeah. I ended up going with collect and instrument. The part that I found most useful within was the {:ns (all-ns)} options to collect!. Didn’t realize you could do that and was worried I was going to have to include a call to collect! in every namespace that used function schemas.#2023-06-2813:32mafcocincoThanks for you help @U055NJ5CC.#2023-09-2214:57eval2020I recently released https://github.com/eval/malli-select to allow for spec2-like selection of malli-schemas. Feedback welcome!#2023-06-2608:25steveb8nrandom thought: Malli defn schemas are perfect for ChatGPT function calling. anyone playing with this yet?#2023-06-2609:12AkizHow should i add another value into :enum?
(def Schema [:map [:description [:and [:map [:subtypes [:enum :subtype1 :subtype2]]] ...]]])
This doesn’t work but should give idea what i am trying to achieve (mu/update-in Schema [:description 0 :subtypes] #(conj % :subtype3)) .#2023-06-2609:22AkizThis works but doesn’t look like a nicest solution to me 😉
(mu/update-in Schema [:description 0 :subtypes] (fn [s] (malli.core/from-ast (update (malli.core/ast s) :values #(conj % :subtype2)))))#2023-06-2609:27p-himik(conj (m/form s) :subtype3)?
FWIW, I'd probably create a factory function and feed it any extra data. So instead of updating a schema at some hard-to-maintain path, you construct it at a single place.#2023-06-2609:42AkizThanks! What you suggest is the plan :)#2023-06-2709:18ikitommithis works too:
(mu/update-in
Schema
[:description 0 :subtypes]
(fn [s] (m/-set-children s (conj (m/children s) :subtype3))))
#2023-06-2709:20ikitommior:
(defn -conj-child [schema value]
(m/-set-children schema (conj (m/-children schema) value)))
+
(mu/update-in Schema [:description 0 :subtypes] -conj-child :subtype3)
#2023-06-2709:35ikitommibut agree, deeply nested updates with schema path can be fragile (wrapping something into a :maybe will accumulate extra 0 into the schema path. But - with Malli, you can also update schemas based on path in value - e.g. just what is visible path in the schemas values itself. for this, there are mu/in->paths and mu/path->in helpers:
(def Schema
[:map [:description [:and [:maybe [:map [:subtypes [:enum :subtype1 :subtype2]]]]]]])
(defn -conj-child [schema value]
(m/-set-children schema (conj (m/-children schema) value)))
(defn -update-in-path
"like update-in, but for value path"
[schema path f & args]
(let [schema (m/schema schema)]
(mu/update-in
schema
(first (mu/in->paths schema path))
(fn [s] (apply f s args)))))
+
(-update-in-path Schema [:description :subtypes] -conj-child :subtype3)
; => [:map [:description [:and [:maybe [:map [:subtypes [:enum :subtype1 :subtype2 :subtype3]]]]]]]
… opposed to:
(mu/update-in Schema [:description 0 0 :subtypes] -conj-child :subtype3)
; => [:map [:description [:and [:maybe [:map [:subtypes [:enum :subtype1 :subtype2 :subtype3]]]]]]]
… not suggesting this a primary way of doing things (`mu/in->paths` can return multiple hits), but might be useful in some contexts.#2023-06-2709:39p-himikNeat!#2023-06-2715:16Akiz🙏#2023-06-2709:35ikitommibut agree, deeply nested updates with schema path can be fragile (wrapping something into a :maybe will accumulate extra 0 into the schema path. But - with Malli, you can also update schemas based on path in value - e.g. just what is visible path in the schemas values itself. for this, there are mu/in->paths and mu/path->in helpers:
(def Schema
[:map [:description [:and [:maybe [:map [:subtypes [:enum :subtype1 :subtype2]]]]]]])
(defn -conj-child [schema value]
(m/-set-children schema (conj (m/-children schema) value)))
(defn -update-in-path
"like update-in, but for value path"
[schema path f & args]
(let [schema (m/schema schema)]
(mu/update-in
schema
(first (mu/in->paths schema path))
(fn [s] (apply f s args)))))
+
(-update-in-path Schema [:description :subtypes] -conj-child :subtype3)
; => [:map [:description [:and [:maybe [:map [:subtypes [:enum :subtype1 :subtype2 :subtype3]]]]]]]
… opposed to:
(mu/update-in Schema [:description 0 0 :subtypes] -conj-child :subtype3)
; => [:map [:description [:and [:maybe [:map [:subtypes [:enum :subtype1 :subtype2 :subtype3]]]]]]]
… not suggesting this a primary way of doing things (`mu/in->paths` can return multiple hits), but might be useful in some contexts.#2023-06-3010:59rickheereHi when I run my application from the repl it all works fine but when I make and run an uberjar build I get an java.lang.NoClassDefFoundError: malli/core/Schema error.#2023-06-3010:59rickheereat integrant.core$build_exception.invokeStatic(core.cljc:285)
at integrant.core$build_exception.invoke(core.cljc:284)
at integrant.core$try_build_action.invokeStatic(core.cljc:296)
at integrant.core$try_build_action.invoke(core.cljc:293)
at integrant.core$build_key.invokeStatic(core.cljc:302)
at integrant.core$build_key.invoke(core.cljc:298)
at clojure.core$partial$fn__5912.invoke(core.clj:2656)
at clojure.core.protocols$fn__8249.invokeStatic(protocols.clj:168)
at clojure.core.protocols$fn__8249.invoke(protocols.clj:124)
at clojure.core.protocols$fn__8204$G__8199__8213.invoke(protocols.clj:19)
at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:31)
at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$reduce.invoke(core.clj:6868)
at integrant.core$build.invokeStatic(core.cljc:325)
at integrant.core$build.invoke(core.cljc:305)
at integrant.core$init.invokeStatic(core.cljc:431)
at integrant.core$init.invoke(core.cljc:423)
at integrant.core$init.invokeStatic(core.cljc:428)
at integrant.core$init.invoke(core.cljc:423)
at com.arqiver.system.bootstrap$init.invokeStatic(bootstrap.clj:107)
at com.arqiver.system.bootstrap$init.invoke(bootstrap.clj:105)
at com.arqiver.system.bootstrap$run_system_BANG_.invokeStatic(bootstrap.clj:115)
at com.arqiver.system.bootstrap$run_system_BANG_.invoke(bootstrap.clj:113)
at com.arqiver.arqiver_portal$_main.invokeStatic(arqiver_portal.clj:8)
at com.arqiver.arqiver_portal$_main.doInvoke(arqiver_portal.clj:6)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at com.arqiver.arqiver_portal.main(Unknown Source)
Caused by: java.lang.NoClassDefFoundError: malli/core/Schema
at malli.util$update_properties.invokeStatic(util.cljc:112)
at malli.util$update_properties.doInvoke(util.cljc:108)
at clojure.lang.RestFn.invoke(RestFn.java:464)
at malli.util$closed_schema$fn__18279.invoke(util.cljc:130)
at malli.core$schema_walker$fn__13649.invoke(core.cljc:2364)
at malli.core$walk$reify__13565._outer(core.cljc:2107)
at malli.core$_walk_entries.invokeStatic(core.cljc:324)
at malli.core$_walk_entries.invoke(core.cljc:322)
at malli.core$_map_schema$reify$reify__12989._walk(core.cljc:1078)
at malli.core$walk.invokeStatic(core.cljc:2102)
at malli.core$walk.invoke(core.cljc:2095)
at malli.util$closed_schema.invokeStatic(util.cljc:125)
at malli.util$closed_schema.invoke(util.cljc:114)
at reitit.coercion.malli$create$fn__20516.invoke(malli.cljc:229)
at reitit.coercion.malli$create$reify__20521._compile_model(malli.cljc:262)
at reitit.ring$_compile_coercion$fn__21191.invoke(ring.cljc:50)
at reitit.impl$_path_vals$_path_vals__19747$fn__19749.invoke(impl.cljc:31)
at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
at clojure.core.protocols$fn__8230.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8230.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$reduce.invoke(core.clj:6868)
at reitit.impl$_path_vals$_path_vals__19747.invoke(impl.cljc:26)
at reitit.impl$_path_vals$_path_vals__19747$fn__19749.invoke(impl.cljc:32)
at clojure.core.protocols$iter_reduce.invokeStatic(protocols.clj:49)
at clojure.core.protocols$fn__8230.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8230.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$reduce.invoke(core.clj:6868)
at reitit.impl$_path_vals$_path_vals__19747.invoke(impl.cljc:26)
at reitit.impl$_path_vals.invokeStatic(impl.cljc:35)
at reitit.impl$_path_vals.invoke(impl.cljc:24)
at reitit.impl$path_update.invokeStatic(impl.cljc:41)
at reitit.impl$path_update.invoke(impl.cljc:40)
at reitit.ring$_compile_coercion.invokeStatic(ring.cljc:50)
at reitit.ring$_compile_coercion.invoke(ring.cljc:49)
at reitit.ring$compile_result$__GT_endpoint__21204.invoke(ring.cljc:58)
at reitit.ring$compile_result$fn__21211.invoke(ring.cljc:75)
at clojure.lang.PersistentArrayMap.kvreduce(PersistentArrayMap.java:429)
at clojure.core$fn__8525.invokeStatic(core.clj:6908)
at clojure.core$fn__8525.invoke(core.clj:6888)
at clojure.core.protocols$fn__8257$G__8252__8266.invoke(protocols.clj:175)
at clojure.core$reduce_kv.invokeStatic(core.clj:6919)
at clojure.core$reduce_kv.invoke(core.clj:6910)
at reitit.ring$compile_result.invokeStatic(ring.cljc:72)
at reitit.ring$compile_result.invoke(ring.cljc:52)
at reitit.impl$compile_route.invokeStatic(impl.cljc:163)
at reitit.impl$compile_route.invoke(impl.cljc:162)
at reitit.impl$compile_routes$fn__19900.invoke(impl.cljc:166)
at clojure.core$keep$fn__8649.invoke(core.clj:7405)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core.protocols$seq_reduce.invokeStatic(protocols.clj:24)
at clojure.core.protocols$fn__8236.invokeStatic(protocols.clj:75)
at clojure.core.protocols$fn__8236.invoke(protocols.clj:75)
at clojure.core.protocols$fn__8178$G__8173__8191.invoke(protocols.clj:13)
at clojure.core$reduce.invokeStatic(core.clj:6886)
at clojure.core$into.invokeStatic(core.clj:6958)
at clojure.core$into.invoke(core.clj:6950)
at reitit.impl$compile_routes.invokeStatic(impl.cljc:166)
at reitit.impl$compile_routes.invoke(impl.cljc:165)
at reitit.core$router.invokeStatic(core.cljc:342)
at reitit.core$router.invoke(core.cljc:313)
at reitit.ring$router.invokeStatic(ring.cljc:128)
at reitit.ring$router.invoke(ring.cljc:98)
at com.arqiver.portal.server$create_app.invokeStatic(server.clj:95)
at com.arqiver.portal.server$create_app.invoke(server.clj:92)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Var.applyTo(Var.java:705)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$apply.invoke(core.clj:662)
at com.arqiver.system.bootstrap$fn__25853.invokeStatic(bootstrap.clj:68)
at com.arqiver.system.bootstrap$fn__25853.invoke(bootstrap.clj:67)
at clojure.lang.MultiFn.invoke(MultiFn.java:234)
at integrant.core$try_build_action.invokeStatic(core.cljc:294)
... 29 more
Caused by: java.lang.ClassNotFoundException: malli.core.Schema
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
... 109 more
#2023-06-3011:00rickheereWhen I strip all the _:coercion_ reitit.coercion.malli/coercion code from my routs the build works. I don't know where to start looking to fix this problem.#2023-06-3011:05delaguardoI would first verify that malli is included into uberjar:
jar -tf /path/to/uber.jar | grep malli
most likely it isn't there
then you should fix the way how you build uberjar#2023-06-3011:12rickheerehere is the full output#2023-06-3011:13rickheere:deps {org.clojure/clojure {:mvn/version "1.11.1"}
org.clj-commons/digest {:mvn/version "1.4.100"}
integrant/integrant {:mvn/version "0.8.0"}
integrant/repl {:mvn/version "0.3.2"}
;; logging
org.clojure/tools.logging {:mvn/version "1.1.0"}
ch.qos.logback/logback-classic {:mvn/version "1.2.3"}
ch.qos.logback/logback-core {:mvn/version "1.2.3"}
org.slf4j/slf4j-api {:mvn/version "1.7.30"}
metosin/malli {:mvn/version "0.11.0"}
com.xtdb/xtdb-core {:mvn/version "1.23.3"}
com.xtdb/xtdb-kafka {:mvn/version "1.23.3"}
com.xtdb/xtdb-s3 {:mvn/version "1.23.3"}
com.xtdb/xtdb-rocksdb {:mvn/version "1.23.3"}
org.apache.kafka/kafka-clients {:mvn/version "2.6.1"}
com.cognitect.aws/api {:mvn/version "0.8.656"}
com.cognitect.aws/endpoints {:mvn/version "1.1.12.415"}
com.cognitect.aws/s3 {:mvn/version "825.2.1250.0"}
com.cognitect/transit-clj {:mvn/version "1.0.333"}
metosin/reitit {:mvn/version "0.7.0-alpha5"}
metosin/reitit-ring {:mvn/version "0.7.0-alpha5"}
ring/ring-jetty-adapter {:mvn/version "1.9.6"}
ring/ring-defaults {:mvn/version "0.3.4"}
hiccup/hiccup {:mvn/version "1.0.5"}
djblue/portal {:mvn/version "0.38.0"}
thheller/shadow-cljs {:mvn/version "2.23.3"}}
#2023-06-3011:18delaguardoDo you explicitly require malli.core somewhere in your code?#2023-06-3011:21rickheereYes I use it in some places for something else then coercion#2023-07-0305:41rickheereI'm still stuck on this. Does anyone have any ideas?#2023-07-0408:20rickheereThis is now solved. I got lost on the Malli error but it turned out to be something else. When that other thing was fixed the error disappeared.#2023-07-0102:09steveb8nQ: I'm using defn schemas with fns that return core.async channels. I wonder if there are any good techniques for validation of data in the channel?#2023-07-0107:57valtteriMy first thought is that validation should probably be done when you put/take#2023-07-0107:59valtteriAnother thought is that maybe you could attach schemas as metadata to the channel?#2023-07-0108:00valtteriOn the producer side.. then the consumers would get that information when they have the chan#2023-07-0108:53delaguardoThe channel can also be created with an attached transducer to take care of in and out record validation.#2023-07-0108:54steveb8nI thought about a channel with a transducer that validates. The easy option is to validate the output in the calling function where the take happens#2023-07-0108:55steveb8nSimultaneous idea 💡#2023-07-0108:56steveb8nThanks to you both for replying#2023-07-0204:32steveb8nThe trick will be how to have the transducer active or not based in m/instrument. If I figure something else I'll create a doc PR#2023-07-0114:41roklenarcicI am trying to model:
<xs:choice>
<xs:element name="taxNumber" type="TaxNumberType" minOccurs="0"/>
<xs:element name="vatNumber" type="VatNumberType" minOccurs="0"/>
</xs:choice>
This is inside a map, so I am writing [:map …. ] schema. How do I model this or statement?#2023-07-0115:05p-himikI'd write it as
[:or
[:map {:closed true}
[:taxNumber {:optional true} ...]]
[:map {:closed true}
[:vatNumber {:optional true} ...]]]#2023-07-0115:14roklenarcicSo specify whole map twice with just one key swapped#2023-07-0115:15p-himikYou can use :and for common parts (but then the :closed part won't work). Or move them in let` and use that binding twice.#2023-07-0115:16roklenarcicprobably that second one, thanks#2023-07-0517:29ikitommion vacation, but now back with the computer - will check all the PRs this week. Unrelated: would like to add a shortcut syntax to lite syntax to support optional keys: if a keyword key starts with ?, it’s optional. Similar to what TypeScript has:
interface User {
name: string;
age?: number;
occupation?: string;
}
=>
(def User
(l/schema
{:name :string
:?age :int
:occupation :string}))
User
;[:map
; [:name :string]
; [:age {:optional true} :int]
; [:occupation :string]]
What do you think?#2023-07-0517:30ikitommiusing ? as a special marker as last char would clash with the idiom of boolean function.#2023-07-0517:31ikitommi, but, one could always force a key with ? in it:
(l/schema
{:name :string
(l/raw :?age) :int
:occupation :string})
;[:map
; [:name :string]
; [:?age :int]
; [:occupation :string]]#2023-07-0517:33ikitommialso, could do a lite->malli converter.#2023-07-0517:38ikitommiwhy?
1. 👍 optional keys in reitit parameters are common => simplifies these a lot
2. 🙇 want to pull some of the good parts of TS to malli
3. 💪 just in lite, e.g. no sugar or design compromises into the malli core lib #2023-07-0517:44teodorluThe :? keyword might be an interesting edge case, predicate or optional?? 😅#2023-07-0517:49ikitommiGood point, it should be just a keyword as the stripped value ( : ) is invalid kw.#2023-07-0518:08isak:?caliente?#2023-07-0518:09isakThat is an example where it would look a little goofy. (The problem is ? is an idiomatic suffix in clojure for booleans.)#2023-07-0518:09isakWhat about putting appending the ? to the type instead, e.g.:
(def User
(l/schema
{:name :string
:age :int?
:occupation :string}))
#2023-07-0520:50Ben SlessIf anything it belongs on the key. The key itself is optional, not the value.
The prefix syntax is fine but I'd put its interpretation behind an option (default off) to avoid collisions by default.#2023-07-0521:07didibusOptions on lite syntax would start to make it feel heavy for me.
I think having it as a prefix + the raw wrapper in case you actually have a keyword that starts with ? is fine. Sure, it's a bit weird to see: :?registered? but not that weird.
Alternatively, could you have a ? wrapper instead:
(l/schema
{:name :string
(l/? :age) :int
:occupation :string})#2023-07-0521:44pithylessI'm wary of using special suffixes or prefixes in the name to signify semantic changes (even for the lite syntax). Considering the lite map syntax only supports keywords as valid [:map] keys, perhaps we can take advantage of this? eg. tuples to unambiguously identify and pass optional data?
(l/schema
{:name :string
[:? :age] :int
:occupation :string})
It's not much more typing than :?age but shows that there is something special about the key itself.#2023-07-0600:20hifumi123Agreed on putting ? on the key, not value. When I see :int?, my mind wants to interpret [:maybe :int] instead of optional key#2023-07-0602:21isakTo me the possibility of already having a ? suffix in the key already is a fatal flaw to the idea of having it as a prefix. Having it on both sides would just be completely nuts. But I can see why having it in the type name is no good either.
What about having a delimiter in the key name (such as . ) that is not commonly used in key names, so it is :age.? :int ? Hiccup uses that idea to add classes and ids to html elements, maybe malli could use it.#2023-07-0606:41ikitommi• there is already l/optional wrapper for value on lite syntax
• there could be wrappers for keys l/req and l/opt instead
• mostly agree that any interpretation of keyword key names should be optional and not enabled by default
• I think lite syntax is not restricted to keywords, any key can be used (did not verify this) -> [:? :age] can also clash
• this boils to simple vs easy :thinking_face: #2023-07-0606:48pithyless> • there could be wrappers for keys l/req and l/opt instead
👍 This would be consistent with existing lite syntax usage.#2023-07-0606:56pithylessMy concern with bringing in more different kinds of code interpretation is that we are increasing the surface area needed to comprehend what the code is doing.
The argument of "let's hide it behind an optional flag" is not convincing to me, because it means in practice there will be codebases that use it and others that don't.
(1) if you walk up to the codebase that chose to use malli, you are forced to understand and learn it. Malli is already a lot to take in at once; malli-lite adds even more cognitive load (since it is different enough from regular malli schemas); I really would rather not have to force someone to also discover that keywords are magically interpreted.
(2) since it's optional, not every malli codebase will have it. So it is even less likely it will be familiar to people walking up to codebases (even if they think they understand malli) :)#2023-07-0607:02didibusDoes the lite syntax already has combination of options like that?
I feel options that alter the syntax of the lite syntax means it's no longer just one syntax, but a combinatronic explosion of possible syntaxes based on the current options. That's neither simple nor easy anymore.
I'd vote for l/req and l/opt, they reduce the amount of typing and visually how much space they take as compared to l/optional, will work for all type of keys, not just keywords, have no conflict with the value of the key, are probably faster to parse, and are pretty self explanatory when you see them.
I think a question mark in front or somewhere in the keyword actually isn't "easy" either, nor is it simple. It's more what we call "convenient" but only for people that know about it and are used to it. Otherwise it also seems to create some inconveniences, like if it clashes with the actual value, like how it can't be used on all types, etc.#2023-07-0613:08ikitommiBased on my experiences, 90% times any added sugar bites back. explicit > implicit, ambiguity issues with syntax are usually really bad. That said, don’t know why I’m tempted with this. Sugar addiction?#2023-07-0613:10ikitommi;; 1) this is the way
(l/schema
{:name :string
(l/opt :age) :int
:occupation :string})
;[:map
; [:name :string]
; [:age {:optional true} :int]
; [:occupation :string]]
;; 2) no surprises
(l/schema
{:name :string
:?age :int
:occupation :string})
;[:map
; [:name :string]
; [:?age :int]
; [:occupation :string]]
;; 3) 🥧🥤
(l/schema
{:name :string
:?age :int
:occupation :string}
{::l/optional-key-syntax true})
;[:map
; [:name :string]
; [:age {:optional true} :int]
; [:occupation :string]]
;; 4) should this be deprecated?
(l/schema
{:name :string
:age (l/optional :int)
:occupation :string})
;[:map
; [:name :string]
; [:age {:optional true} :int]
; [:occupation :string]]#2023-07-0613:12Ben Sless1 makes sense, but since the theme is lite syntax, how about just calling the operator ??#2023-07-0613:12ikitommiyou mean:
(l/schema
{:name :string
(l/? :age) :int
:occupation :string})#2023-07-0613:13Ben Slessyes#2023-07-0616:41didibusMaybe the popularity of this question teaches us something? https://stackoverflow.com/questions/37632760/what-is-the-question-mark-for-in-a-typescript-parameter-name#2023-07-0600:59didibusSeems cljdoc is failing to show the API ref: https://app.circleci.com/pipelines/github/cljdoc/builder/41462/workflows/711cd569-326e-4497-ae5a-f03fe0b44148/jobs/57837#2023-07-0603:52joshchoAre there ways to provide custom anonymous function wrapper for Malli schemas? I am looking at https://github.com/CrypticButter/snoop for the defn wrapper. I am imagining syntax like the following:
(>fn [x y]
[int? int? => int?]
x + y)
Main use case for me is in re-frame, where large anonymous functions are frequently used for registering events. Perhaps this is just a simple macro away, but I am wondering if a library may exist.#2023-07-0606:28ikitommiSee m/-instrument. No macro/sugar on top of that atm. Could add mx/fn for this.#2023-07-0606:28ikitommialso, check https://github.com/teknql/aave#2023-07-0606:29ikitommiMaybe request this in snoop?#2023-07-0706:20Sathiya PriyanHi, I am running into some trouble using malli registry options.
(m/validate
[:schema {:registry {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]}}
[:ref ::cons]]
[16 [64 [26 [1 [13 nil]]]]])
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
:malli.core/child-error
Can some one help with this. It seems to be working with some versions of malli. But with the newer versions its giving errors. I am not sure what the issue#2023-07-0706:23p-himikWorks just fine for me on 0.11.0.#2023-07-0706:31Sathiya Priyani get the same error on 0.11.0 too.#2023-07-0706:32Sathiya Priyanshould i upgrade any other libraries too to avoid this#2023-07-0706:32Sathiya Priyan#error {
:cause ":malli.core/child-error"
:data {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :schema, :properties {:registry #:dev{:cons [:maybe [:tuple pos-int? [:ref :dev/cons]]]}}, :children [[:ref :dev/cons]], :min 0, :max 0}}
:via
[{:type clojure.lang.ExceptionInfo
:message ":malli.core/child-error"
:data {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :schema, :properties {:registry #:dev{:cons [:maybe [:tuple pos-int? [:ref :dev/cons]]]}}, :children [[:ref :dev/cons]], :min 0, :max 0}}
:at [malli.core$_exception invokeStatic "core.cljc" 138]}]
:trace
[[malli.core$_exception invokeStatic "core.cljc" 138]
[malli.core$_exception invoke "core.cljc" 136]
[malli.core$_fail_BANG_ invokeStatic "core.cljc" 142]
[malli.core$_fail_BANG_ invoke "core.cljc" 140]
[malli.core$_check_children_BANG_ invokeStatic "core.cljc" 168]
[malli.core$_check_children_BANG_ invoke "core.cljc" 160]
[malli.core$_simple_schema$reify__3685 _into_schema "core.cljc" 644]
[malli.core$into_schema invokeStatic "core.cljc" 1992]
[malli.core$into_schema invoke "core.cljc" 1983]
[malli.core$schema invokeStatic "core.cljc" 2052]
[malli.core$schema invoke "core.cljc" 2034]
[malli.core$validator invokeStatic "core.cljc" 2116]
[malli.core$validator invoke "core.cljc" 2110]
[malli.core$validate invokeStatic "core.cljc" 2124]
[malli.core$validate invoke "core.cljc" 2118]
[malli.core$validate invokeStatic "core.cljc" 2122]
[malli.core$validate invoke "core.cljc" 2118]
[dev$eval49915 invokeStatic "form-init3374836250330622890.clj" 1]
[dev$eval49915 invoke "form-init3374836250330622890.clj" 1]
[clojure.lang.Compiler eval "Compiler.java" 7194]
[clojure.lang.Compiler eval "Compiler.java" 7149]
[clojure.core$eval invokeStatic "core.clj" 3215]
[clojure.core$eval invoke "core.clj" 3211]
[nrepl.middleware.interruptible_eval$evaluate$fn__48754$fn__48755 invoke "interruptible_eval.clj" 87]
[clojure.lang.AFn applyToHelper "AFn.java" 152]
[clojure.lang.AFn applyTo "AFn.java" 144]
[clojure.core$apply invokeStatic "core.clj" 667]
[clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1990]
[clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1990]
[clojure.lang.RestFn invoke "RestFn.java" 425]
[nrepl.middleware.interruptible_eval$evaluate$fn__48754 invoke "interruptible_eval.clj" 87]
[clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
[clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
[clojure.main$repl$fn__9215 invoke "main.clj" 458]
[clojure.main$repl invokeStatic "main.clj" 458]
[clojure.main$repl doInvoke "main.clj" 368]
[clojure.lang.RestFn invoke "RestFn.java" 1523]
[nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 84]
[nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 56]
[nrepl.middleware.interruptible_eval$interruptible_eval$fn__48787$fn__48791 invoke "interruptible_eval.clj" 152]
[clojure.lang.AFn run "AFn.java" 22]
[nrepl.middleware.session$session_exec$main_loop__48857$fn__48861 invoke "session.clj" 218]
[nrepl.middleware.session$session_exec$main_loop__48857 invoke "session.clj" 217]
[clojure.lang.AFn run "AFn.java" 22]
[java.lang.Thread run "Thread.java" 829]]}
This is the stack trace#2023-07-0706:52p-himikWhere did that :min 0, :max 0 part come from?#2023-07-0706:54p-himikAh, those are the default values in -simple-schema.#2023-07-0706:55p-himikDoes the error also appear when you check the above schema in a REPL started with clj -Sdeps '{:deps {metosin/malli {:mvn/version "0.11.0"}}}'?#2023-07-0707:40Sathiya Priyansorry. i think the issue was in some other place due to invalid schemas.#2023-07-0707:40Sathiya Priyanthansk for your help @U2FRKM4TW#2023-07-1019:41radsAre there any concise function-level API docs available for Malli? I noticed there are docstrings on most functions but they aren't exposed directly in the docs: https://cljdoc.org/d/metosin/malli/0.11.0/doc/readme
(the namespace docs say "API analysis failed")#2023-07-1019:42radsAh, my bad, I see this was posted earlier: https://clojurians.slack.com/archives/CLDK6MFMK/p1688605145144229#2023-07-1309:03Ngoc KhuatHi there, do we have any plan to add support defmethod inline schemas?#2023-08-0817:15ikitommiHow would that work? Each dedmethod would have unique inputs? Or just return values?#2023-07-1319:26theequalizer73Hi Folks, I’m using reitit and malli, and I’m trying to validate/coerce a response from the DB. I’ve tried with malli.core/schema and malli.experimental.lite/schema using something like
(m/schema
[:map
[:key1 :int]
[:key2 :double]])
and also
(l/schema
{:key1 int?
:key2 double?})
From the DB I’m getting java.lang.Long and java.lang.Double but I’m getting a 500 error with this
{:type "reitit.coercion/response-coercion",
:coercion "malli",
:in ["response" "body"],
:humanized {:key1 ["should be an int"],
:key2 ["should be a double"]}
I don’t want to parse anything. I just need to check the types from the DB and even when I’m clearly getting something like 38 or 7.99 I can’t pass the coerce response. Any hints on how to solve this?#2023-07-1412:37theequalizer73I’m blocked with this ^ anyone?#2023-07-1412:57rafaeldelbonitry with number?#2023-07-1412:58rafaeldelboniwe did this when dealing with doubles
https://github.com/parenthesin/microservice-boilerplate-malli/blob/main/src/microservice_boilerplate/schemas/db.clj#2023-07-1413:02theequalizer73Thanks, I will try number? but no rigmarole, this should be simple. I was planning to go with clojure.spec.alpha today b/c this is taking me too much time.#2023-07-1413:06rafaeldelbonischema is a nice alternative as well https://github.com/plumatic/schema super simple#2023-07-1413:13theequalizer73Thanks#2023-07-1413:19rafaeldelbonitil the word rigmarole 😄#2023-07-1414:23theequalizer73I’ve found the problem. The DB is actually returning java.math.BigDecimal and java.lang.Float . I think I’m gonna def my own version of int and double. BTW number? and float? should work!#2023-07-1415:14Ben SlessYou can add decoders, too#2023-07-1415:15Ben SlessAnd which db returns float?#2023-07-1415:18theequalizer73This is a very particular thing, but now that I now where the problem is I think Imma add my own predicates or use rational?#2023-07-1803:17fabraoHello all, why it seems that when I use :optional true the validation function does not work. Like
(->
(mc/explain
[:map [:good {:optional true} [:vector [:map [:id :string]]]
[:fn {:error/message "Wrong validation"}
(fn [x] false)]]]
{:good [{:id "1"} {:id "2"}]})
(me/humanize)) => nil
(->
(mc/explain
[:map [:good [:vector [:map [:id :string]]]
[:fn {:error/message "Wrong validation"}
(fn [x] false)]]]
{:good [{:id "1"} {:id "2"}]})
(me/humanize)) => {:good ["Wrong validation"]}#2023-07-1807:42JacquesYou need to :and your :map and :fn. :fn shouldn't be nested under the :map:
(->
(m/explain
[:and
[:map [:good {:optional true} [:vector [:map [:id :string]]]]]
[:fn {:error/message "Wrong validation"}
(fn [x] false)]]
{:good [{:id "1"} {:id "2"}]})
(me/humanize))#2023-07-1813:58fabraothank you, I'll go with this for more "in-line":
(->
(malli.core/explain
[:and
[:map [:good {:optional true} [:and
[:vector [:map [:id :string]]]
[:fn {:error/message "Wrong validation"}
(fn [_] false)]]]]]
{:good [{:id "1"} {:id "2"}]})
(me/humanize))#2023-07-1806:52Karol WójcikI saw this PR which looks extremely useful. I stumbled upon the same problem many times.
Any objections from the core team to push it upstream?
https://github.com/metosin/malli/pull/897#2023-08-0817:54ikitommiMerged, thanks!#2023-07-1817:08cap10morganIn a map schema, what's the best way to say that one and only one of a set of keys must be present? Setting them all to {:optional true} allows none of them to be present, which I don't want. Just use :fn?#2023-07-1818:03Ben SlessDo some schema transforms and or all of them#2023-07-1818:04cap10morgannot sure I follow. could you elaborate?#2023-07-1818:06Ben SlessLet's say you can only have keys a&b or b&c out of a,b,c
roughly:
(into [:or] (map #(my/select-keys YourMap ks) allowed-sets))#2023-07-1818:07Ben SlessAllowed sets is [[:a :b] [:b :c]]#2023-07-1818:07Ben SlessYourMap [:map a b c ,,,]#2023-07-1818:07Ben SlessKinda gross but works#2023-07-1818:08cap10morganah, I see. I meant there's one set of keys, and the map must contain one and only one of the keys#2023-07-1819:20Stig BrautasetIt doesn’t look like :map has a max/min size, so probably need an extra :fn schema. Also probably close the schema so you’re sure that it’s one of your defined keys in the schema, and not a rogue one: https://malli.io/?value=%7B%3Afoo%201%7D&schema=%5B%3Aand%0A%20%5B%3Amap%20%7B%3Aclosed%20true%7D%0A%20%20%5B%3Afoo%20%7B%3Aoptional%20true%7D%20%3Aint%5D%0A%20%20%5B%3Abar%20%7B%3Aoptional%20true%7D%20%3Aint%5D%5D%0A%20%5B%3Afn%20(fn%20%5Bm%5D%20(%3D%20(count%20m)%201))%5D%5D%0A%20#2023-07-1819:28Ben SlessJust let allowed-sets to be [[:a] [:b] [:c]], same thing#2023-07-1819:28Ben Slessfn schema is evil and best avoided#2023-07-1819:31Ben Slessyou also want to close the schema#2023-07-1819:31Ben Sless[:orn [:a [:map {:closed true} [:a A]] ,,,]#2023-07-1819:32Ben Sless^that's another option#2023-07-1820:36hifumi123Very similar question as the one above: is there a declarative way to require a pair of keys? i.e. a map where a pair [:x :y] of keys is either present or absent.
For a concrete example, let’s start with this schema
[:map
[:name string?]
[:x number?]
[:y number?]]
This will allow {} and {:x 1 :y 2}, which is what I want. However, it also allows {:x 1}, which I do not want.
We can use either a multi-schema for fn schema for this. But I would rather not have to do this, because this isnt the only “pair of keys” I will add to my schema.#2023-07-2118:19dvingoI was thinking a clever use of the :pred option to -map-schema could work, but to make it declarative it would be nice to use the properties passed to the schema to describe the desired constraints.
I added a second predicate for map-schemas in my fork of malli as a proof of concept:
https://github.com/dvingo/malli/commit/1f3d00f8abbc0aafede329d558931f110427f3b5
This supports arbitrary logic on the data and keys in the map e.g.:
(def a-schema (m/schema [custom-map-schema
{:strict-keys #{:x :y}}
[:z :int]
[:a {:optional true} :int]
[:x {:optional true} :int]
[:y {:optional true} :int]])
which will fail if :x is present but :y is missing (and vice versa) and will also pass if both are absent.
I think this sort of thing would be generally useful for lots of use-cases besides relations between the keys (anywhere you want key-dependent or value-dependent logic)
There was something related done here
https://github.com/bsless/malli-keys-relations/tree/master
but using :and schema instead of custom :map schemas and is not extensible in this way
you'd also have to handle implementing custom generators for these custom schemas, but that would have to be handled regardless#2023-07-2406:53hifumi123I just realized what I want is very similar in spirit to spec2's select 😄
I would just want
(s/select ::some-schema [::name ::pair {::pair [::x ::y]}])
#2023-07-2412:18dvingooh nice, should be possible using a walker, here is some inspiration:
https://github.com/metosin/malli/blob/15c14e1beacb000b7eb023675e33f0711962fced/src/malli/util.cljc#L114
The tricky thing is dealing with refrences and wrapper schemas (`[:schema]`) when walking in order to map to the appropriate input selection pattern#2023-07-1919:55cap10morganShould this work?
(-> (m/schema ::my-schema {:registry {::my-schema string?}})
m/ast
m/from-ast)
...because it doesn't seem to. throws malli.core/invalid-schema b/c the m/ast call returns {:type :malli.core/schema, :value :full.ns/my-schema}#2023-07-1920:01cap10morgan(pretend I merged in malli.core/predicate-schemas)#2023-07-2116:15dvingoYou need to pass the registry to all of the calls:
(let [opts {:registry (merge (m/default-schemas) {::my-schema string?})}]
(-> (m/schema ::my-schema opts)
(m/ast opts)
(m/from-ast opts)))#2023-07-2117:23dvingoThis also works:
(m/from-ast
(m/ast
[:schema {:registry {::my-schema string?}}
::my-schema]))#2023-07-2118:33cap10morganah, thanks!#2023-07-2118:36cap10morganhmm, that doesn't fix it for me. same error. same return value from m/ast#2023-07-2119:20dvingowhich version? can you post some code?#2023-07-2119:21dvingoboth are working for me#2023-07-2302:01steveb8n@ikitommi https://github.com/metosin/malli#schema-Transformation is a currently dead link in the readme#2023-07-2302:01steveb8nshould be https://github.com/metosin/malli#schema-transformation#2023-07-2501:18Michael Gardneris there a way to make malli.dev.pretty elide certain parts of a schema, like a very large enum?#2023-08-0103:20hifumi123As a quick workaround, does binding *print-length* work for your use case?#2023-08-0716:08Michael Gardnergood suggestion. I'll have to gather feedback about how other devs here feel about it#2023-08-0119:44pavlosmelissinosHi! If you m/explain invalid data using a multi-schema, the explainer shows the entire multi-schema, not just the matching part. This can make spotting errors harder.
Wouldn't it be better to just use the part that is relevant when there's a clear dispatch value? :thinking_face: Example in thread!#2023-08-0119:49pavlosmelissinos(m/explain
[:multi {:dispatch :type}
[:sized [:map [:type keyword?] [:size int?]]]
[:human [:map [:type keyword?] [:name string?] [:address [:map [:country keyword?]]]]]]
{:type :sized, :sizey 10})
If you run this (took it from the malli README and added a typo to the data so that it throws an exception), you'll get a message that tells you that the map is an invalid ":sized OR :human"
I think it would be an improvement if malli could understand that the map is obviously a :sized in this case, so I don't need to be shown any information about :human.#2023-08-0120:01pavlosmelissinosI.e., instead of this:
{:schema [:multi {:dispatch :type} [:sized [:map [:type keyword?] [:size int?]]] [:human [:map [:type keyword?] [:name string?] [:address [:map [:country keyword?]]]]]], :value {:type :sized, :sizey 10}, :errors ({:path [:sized :size], :in [:size], :schema [:map [:type keyword?] [:size int?]], :value nil, :type :malli.core/missing-key})}
I'd prefer this:
{:schema [:sized [:map [:type keyword?] [:size int?]]], :value {:type :sized, :sizey 10}, :errors ({:path [:sized :size], :in [:size], :schema [:map [:type keyword?] [:size int?]], :value nil, :type :malli.core/missing-key})}#2023-08-0308:53ikitommigood catch, I think that’s more correct. please write an issue out of this#2023-08-0308:56pavlosmelissinosThanks for the confirmation! I'm away from my computer right now but I'll make an issue as soon as I get back!#2023-08-0312:55pavlosmelissinosFYI, voila! https://github.com/metosin/malli/issues/923
edit: rephrased it a bit...#2023-08-0123:43JoelHow could I figure out the type of :x? Either from the registry “directly” or after turning into a schema.
(m/schema [:schema {:registry {:x :int}} :x])
#2023-08-0202:38JoelIt appears the relevant function I need is deref#2023-08-0218:45markbastianIs there a way to produce malli schemas from swagger? The inverse of what is found https://github.com/metosin/malli#swagger2 on the main malli page.#2023-08-0219:03Brennan C.I think you're looking for https://github.com/metosin/malli/issues/54 (with link to some work on that front that you may be able to use/improve).#2023-08-0219:04markbastianThanks!#2023-08-0408:17joshkhas i’m m/walking the keys of a map schema, how can i tell if a key is optional? i thought m/properties would have returned the {:optional true} properties map in the example below:
(m/walk
[:map
[:x :int]
[:y {:optional true} :int]]
(m/schema-walker
(fn [s]
(println "props" (m/properties s))
s)))
props nil
props nil
props nil
thanks!#2023-08-0408:28p-himikSchema walker walks schemas. :int is a schema, [:map ...] is a schema, but [:y ...] is not.
Unfortunately, even if you provide a more flexible walker function yourself, you still won't get the props of a key. Not sure what the right solution here is.#2023-08-0408:54joshkhgood point, [:y …] is not a schema. my goal is to wrap all “map value schemas”, like :int above, with a [:or nil %] but only when the map key is optional :thinking_face:. i’m having a look at mu/make-required and that seems to remove :optional from that map using some plain old clojure#2023-08-0408:56p-himikJust in case - if you control the original schema, I'd create a wrapper function that, given :y and :int, outputs [:y {:optional true} [:maybe :int]].#2023-08-0408:59joshkhi’m not sure i follow — i have an existing large map schema, where i want to programatically derive a second schema where optional keys are :maybe. when you say wrapper function, do you mean for the walk?#2023-08-0409:04p-himikI mean that if the existing large map schema is something that you can control, then, instead of modifying an existing schema IMO it would be both easier and more straightforward to just create two different schemas.
(defn make-the-schema [optional-is-nilable?]
(letfn [(opt [k v] [k {:optional true} (if optional-is-nilable? [:maybe v] v)])]
[:map [:x :int] (opt :y :int)]))
(def schema (make-the-schema false))
(def nilable-schema (make-the-schema true))#2023-08-0409:44joshkhoh gotcha. okay, thanks for the feedback and suggestion 🙂 i’ll give it a whirl.#2023-08-0906:54ikitommiFew tips on this:
• m/entries returns a sequence of map entries as (internal) entry schemas. calling m/properties gives the entry properties
• you can call m/children on map and get a 3-tuple of key properties val.
• if you use m/walk, you can add an option ::m/walk-entry-vals to walk also the entries, see https://github.com/metosin/malli/blob/master/docs/tips.md#walking-schema-and-entry-properties#2023-08-0906:55ikitomminot the most elegant api, but works.#2023-08-0414:18JoelI see that :int accepts both ints and longs, I’m just curious if anyone has regretted not being more specific on the type (integers vs. longs or floats vs. doubles), particularly if you are working with Java interop.#2023-08-0418:16Ben SlessIt probably maps on to what int? accepts#2023-08-0419:58hanDerPedergiven this schema:
[:map [:from {:optional true} :time/instant] [:to {:optional true} :time/instant]]
I'd like :to to default to now, and :from to default to now-2hours. Is that possible using mt/default-value-transformer?#2023-08-0420:29hanDerPederThis works:
(m/decode
[:map
[:from {:default :now-2h} :time/instant]
[:to {:default :now} :time/instant]]
{}
(mt/default-value-transformer {:default-fn (fn [_ x ] (case x
:now (t/now)
:now-2h (t/<< (t/now) (t/of-hours 2))
x))}))
but I was hoping I could do something like:
(m/decode
[:map
[:from {:default-fn #(t/<< (t/now) (t/of-hours 2))} :time/instant]
[:to {:default-fn #(t/now)} :time/instant]]
{}
mt/default-value-transformer)#2023-08-0906:05ikitommimaybe add support for :default/fn property?#2023-08-0906:05ikitommithat should be a small change to the default-transformer impl#2023-08-0911:30hanDerPederdone, https://github.com/metosin/malli/pull/927#2023-08-0518:46radhikaquestion: although parsing an array of values using a :catn schema yields a map of named entries, then unparsing the same map with the same :catn schema does not seem to guarantee preserving entry order in the result array.
I guess unparsing consumes the seq of input map entries in its natural unstable order, but is this positional lossiness expected behavior for a sequence schema? (example in thread)#2023-08-0518:47radhikaexample:
> (let [schema-8 [:catn
[:a :int]
[:b :int]
[:c :int]
[:d :int]
[:e :int]
[:f :int]
[:g :int]
[:h :int]]
schema-9 (into schema-8 [[:i :int]])
input-8 [1 2 3 4 5 6 7 8]
input-9 (into input-8 [9])]
{:roundtrip-8 (->> input-8
(m/parse schema-8)
(m/unparse schema-8))
:roundtrip-9 (->> input-9
(m/parse schema-9)
(m/unparse schema-9))})
{:roundtrip-8 [1 2 3 4 5 6 7 8], :roundtrip-9 [5 7 3 8 2 4 6 9 1]}
#2023-08-0520:33ikitommiOh, that's not good. Please write an issue.#2023-08-0521:44radhikasubmitted https://github.com/metosin/malli/issues/925
but I'm not convinced catn behaving like this is necessarily wrong.
I added a workaround to the ticket which could be abstracted into a new type of schema intentionally returning a MapEntry sequence to preserve order rather than an unstable map#2023-08-0905:47ikitommiFixed#2023-08-0614:07mengudo validations such as uniqueness belong to a malli schema? what’s the stance there?
(def User
[:map
[:id :uuid]
[:username [:and :string [:not username-exists?]]]
[:email [:and :string [:not email-exists?]]]
[:password :string]])
feels like it’s not to be honest because the exists? fns also need another argument, db to be able to run a query and I couldn’t figure out a safe way to do that.#2023-08-0614:10menguthis way I know I broke at least the generate fn for example#2023-08-0615:33valtteriTalking to the database should happen outside malli#2023-08-0615:35valtteriThis kind of check where the underlying data is mutable (database) you should make the check in the db most probably.#2023-08-0615:37valtteriIt would be possible to read all the usernames from the db and construct a malli schema that forces uniqueness at that point in time but that would be highly inefficient.#2023-08-0615:39valtteriYour db probably already has a uniqueness constraint and you can catch a possible violation quite easily (outside malli).#2023-08-0616:14mengui think what tempts me here is the standardization of error message generation and validation#2023-08-0616:15menguof course I can let the db to throw constraint exception and catch it there#2023-08-0616:15valtteriYep, it may sound tempting. But you are not going to catch all the possible error situations of your app with malli 🙂#2023-08-0616:16valtteri(at least in most apps you don’t)#2023-08-0616:16menguof course#2023-08-0616:17valtteriSo maybe you can reuse the error reporting from malli with other error messages as well?#2023-08-0616:17valtteriThough you can’t control error messages that are from 3rd party libraries or Clojure itself#2023-08-0616:17valtteriBut I guess you’re talking about errors returned by your API?#2023-08-0616:17valtteriOr dev-time error reporting?#2023-08-0616:17menguerrors returned by my API indeed#2023-08-0616:18menguso, right now, here’s what I’ve come up with:
(def SignupRequest
[:map
[:username [:string {:min 3, :max 100}]]
[:email [:re {:error/message "must be a valid e-mail address."}
#"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"]]
[:password [:string {:min 8}]]])
(defn gen-unique-constraints [db]
[:map
[:email [:fn {:error/message "is already taken."} #(email-available? db %1)]]
[:username [:fn {:error/message "is already taken."} #(username-available? db %1)]]])
(defn is-valid-user? [db posted-user]
(let [unique-schema (gen-unique-constraints db)
unique-user? (->> posted-user (m/explain unique-schema) me/humanize)
valid-request? (->> posted-user (m/explain SignupRequest) me/humanize)]
(merge-with into unique-user? valid-request?)))
so now I get a result like:
{:username [is already taken.], :email [must be a valid e-mail address.]}
#2023-08-0616:19mengui’ve separated the actual request from the second validation but merged the errors into one. not sure if i’m abusing here but i’m not not liking it 😄#2023-08-0616:21valtteriDoesn’t seem like too bad abuse to me. 😉 You can also try your luck with malli.error namespace directly#2023-08-0616:22valtteriTommi (the author) makes everything modular and composable#2023-08-0616:25menguI’ll take a look at that#2023-08-0616:25mengutbh so far I’m contempt 😄#2023-08-0710:12pithylessSome thoughts to consider over coffee:
The DB uniqueness constraints need to be checked atomically, so unless you're doing this all within a locked DB transaction, you still need to check again during the actual DB write, catch the potential errors, and render the user error messages. So probably there is a second piece of code that does something very similar - what does the malli validation beforehand give you?
This looks as if you're trying to repurpose malli into a declarative control-flow library. Once you start adding more and more logic and branching to the control flow, you may realize you want something with more control knobs and hooks than malli can provide.
Just like clojure.spec was repurposed a lot for data coercion (which had lots of rough edges), I think repurposing malli for control-flow also may have some rough edges that are not immediately obvious.#2023-08-0917:10ikitommiAnyone interested in working with expound-like in-place-highlighting-of-errors? quickly tested how the schema-of-schemas would work to get better errors on invalid schema syntax. Pretty colors != human-readable#2023-08-0917:12ikitommie.g. each IntoSchema has defined it’s property and children schemas - this is used to validate the schema syntax at development time.#2023-08-0917:14ikitommialso, the vector syntax doesn’t print nicely with fipp atm, this:
[:catn [:args [:cat [:= :cat] [:* :any]]] [:return :any]]
should be:
[:catn
[:args [:cat
[:= :cat]
[:* :any]]]
[:return :any]]
… I think this would be easy to do.#2023-08-0917:15ikitommialso, special colors for type and entry keys.#2023-08-1222:58steveb8nQ: what is wrong with this…
(let [location-schemas (merge {:address :string} (m/default-schemas))]
(-> [:map
[:job [:map
[:title :string]]]]
(mu/get :job)
(mu/assoc :location :address {:registry location-schemas})))
=>
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
:malli.core/invalid-schema#2023-08-1222:59steveb8nwhen I comment out the (mu/get :job) it works so the local registry is ok#2023-08-1223:00steveb8nI debugged in Cursive and I see that the :options are lost on the job schema during mu/assoc. could this be a bug or am I thinking about this wrong?#2023-08-1223:02steveb8nit seems like mu/assoc doesn’t work if the input schema comes from mu/get#2023-08-1223:27steveb8nhappy to log a bug but would very much appreciate a workaround if there is one#2023-08-1309:15ikitommiPlease file a bug.#2023-08-1309:24steveb8nSounds good. Will do first thing tomorrow#2023-08-1308:10hifumi123I have an application using reitit for routers and swagger docs. My endpoints all respond with data conforming to some malli schema. Is it possible to convert the these schemas into models in the swagger docs?
For a concrete example, I have route data that looks like this
["/thing/:id"
{:get
{:handler get-thing
:responses {200 {:description "Successful response"
:body Thing}
404 {:description "Not found"
:body Anomaly}}}}]
and I would like the Thing and Anomaly schemas to become models in the swagger docs. I am able to get Malli to convert the map schemas into Swagger data, but I am not sure how to embed it into the Swagger UI.#2023-08-1309:17ikitommiNot sure what is the problem. All malli-schemas get converted into swagger schemas out of the box?#2023-08-1309:22hifumi123Sorry. Looks like there isnt a problem. Basically, the demo Swagger UI has a dedicated “models” section and I wanted to achieve this with Malli schemas#2023-08-1309:23hifumi123It turns out this dedicated models section is only available with OpenAPI 3.0, not Swagger 2.0#2023-08-1309:31ikitommiOpenAPI3.x support should land in few weeks 😎#2023-08-1404:07JoelIs there a way to get the type information in regard to a function?
(defn square [x] (* x x))
(m/=> square [:=> {:registry {:a :int}} [:cat :a] nat-int?])
How do I “grab” input/output for square as I’d like to use it elsewhere?#2023-08-1404:13JoelI got as far as (m/function-schemas), but I’d like to ascertain the input/outputs for square#2023-08-1405:23ikitommiYou can ask m/children on the schema or call m/ast on it and use the named keys on the result (I recall :input & :output.#2023-08-1405:24ikitommiCurious, what is your use case here?#2023-08-1413:59JoelI’m using Malli to generate data for Pathom, in particular the inputs/outputs in this case. So what I’m really interested in is the registry keywords.#2023-08-1414:19JoelTo look-up square in the map, this works, but I’m guessing there’s a better “trick”?
(get (get (m/function-schemas) (symbol (namespace `square))) 'square)
#2023-08-1406:47ikitommiDoes someone have good arguments for or against the correctness on how :or is decoded now: https://github.com/metosin/malli/issues/910#issuecomment-1676768588#2023-08-1501:21steveb8nI would like to comment but haven’t been affected by this so cannot without more context. if you provide context, I’ll try to add my opinion#2023-08-1504:14jasonjcknmy 2 cents
I would avoid #1 , because it's not very nice mathematically
(def T (mt/transformer {:name :math}))
(def S1 [:map {:closed true, :decode/math #(assoc % :b 'bad-input)}
[:a :int]
[:b :int]])
(def S2 [:map {:closed true, :decode/math #(assoc % :a 'bad-input)}
[:a :int]
[:b :int]])
(def I {:a 4, :b 5})
(m/decode S1 I T) ;=> {:a 4, :b bad-input}
(m/decode S2 I T) ;= {:b 4, :a bad-input}
(m/decode [:or S1 S2] I T) ;=> {:a 4, :b 5}
S1 would never validate value I transformed
S2 would never validate value I transformed
Yet the S1 ∪ S2 validates value I ‘transformed’ by virtue of the ‘transformation’ being a no-op
I may have a use case that's too contrived to explain, but in theory it could hit this, (in practice probably not), and #1 would be the wrong choice.
I like both #2 and #3, my intuition slightly prefers #3. As a side note, could you just produce an error, is that a 4th choice.
In stricter functional implementations of these 'bijection-like' transformations, when you transform from type A into type B, the return value is either (1) of type B or (2) an error of some kind, the untransformed values are bit dangerous, as it can accidentally get validated as B, when in fact untransformed values of type A just look a bit like B in pathological cases, if that makes sense .#2023-08-1509:19ikitommiThanks @U0J3J79FE for the long answer! Few comments:
• goal of decode is to produce valid values, so all cases where decode breaks it are kinda wrong
• I agree on 1 being wrong
• m/coerce does both decode & validate
• I'm also leaning also towards 3 (today)#2023-08-1510:32jahsonHi! Looks like after https://github.com/metosin/malli/blob/master/CHANGELOG.md#0103-2023-03-18 mt/strip-extra-keys-transformer started to strip all keys from :map if there is no ::m/default provided.
(m/decode
[:map]
{1 1, 2 "2", "3" 3, "4" "4"}
(mt/strip-extra-keys-transformer))
; => {}
Is it working as intended?#2023-08-1513:42Noah Bogartif you don't mind, how do you expect it to work?#2023-08-1513:56jahsonI never thought about it, actually.#2023-08-1513:57jahsonAnd I might be biased by the way it worked before.#2023-08-1513:58Noah Bogarthow did it work before?#2023-08-1513:58jahsonActually, seems like I don’t really know 🙃#2023-08-1513:59jahsonTrying to reproduce )#2023-08-1513:59Noah Bogarthaha classic#2023-08-1513:59Noah Bogartthat's annoying#2023-08-1513:59jahsonOkay, got it.#2023-08-1514:00jahson(m/decode
[:map]
{1 1, 2 "2", "3" 3, "4" "4"}
(mt/strip-extra-keys-transformer))
; => {1 1, 2 "2", "3" 3, "4" "4"}#2023-08-1514:00jahsonSeems like my memory not failed me this time.#2023-08-1514:01jahson[metosin/malli "0.10.2"]#2023-08-1514:01Noah Bogarti'm asking because to me, stripping all keys is the only way i can make sense of "map schema with no keys" plus "strip extra keys". if there are no defined keys, then every key is extra. for validation, :map is completely open. but for stripping (coercion), idk if i agree#2023-08-1514:08jahsonThat is one way to think about it. The other way might be to think of [:map] as an open map that might contain any key with any value. This way you get a map that might evolve for some time, before it could take final form.
But as I’ve said, I might be biased, because it just took a couple of hours of my life )#2023-08-1514:37Noah Bogartyeah for sure, i don't mean to discount the effort#2023-08-1514:45jahsonAnyway, this change isn’t marked as breaking in changelog and my intent was to know if this effect was intended, and if it was, then propose to mark is as breaking.#2023-08-1514:55Noah BogartI bet the folks at metosin would be willing to do that for you. they're responsive on github#2023-08-1516:14ikitommiLooks fishy. Just ending my 9 week vacation, have no memory of why that is changed. Looking at the CHANGELOG:
* mt/strip-extra-keys-transformer works with :map-of.
might have changed the behavior of :map unintentionally. Please write an issue of this. I’ll check the commit messages if there is clue for this.#2023-08-1516:17ikitommihttps://github.com/metosin/malli/pull/871/commits/0e5b5e3ac56479f75c812c1320ac1f1ebfba8f76#2023-08-1516:18ikitommi(testing "extra keys from :map are stripped"
(is (= {:x 1, :y 2}
(m/decode
[:map [:x :int] [:y :int]]
{:x 1, :y 2, :z 3}
(mt/strip-extra-keys-transformer)))))#2023-08-1516:18ikitommi:thinking_face:#2023-08-1516:19ikitommias maps are open by default, I think it’s bit odd to strip extra keys. For closed maps, sure.#2023-08-1521:12jahsonAnother side effect of this change is when you merge two map definitions and the first one has default key:
(def registry (merge (m/default-schemas) (mu/schemas)))
(def Merged
(m/schema
[:merge
[:map
[:x :string]
[::m/default [:map-of :keyword :any]]]
[:map {:closed true}
[:y :int]]]
{:registry registry}))
(m/validate Merged {:x "kikka", :y 6, :z "invalid"})
; => true
I.e. map is marked as closed, but it is not really closed.#2023-08-1603:53JoelI’m able to tell if the map uses keywords:
(mu/find-first [:map
[:item-a string?]
[:misc [:vector string?]]]
(fn [schema _ _]
(some keyword? (-> schema mu/keys))))
But how do i do this if the map is part of a registry?
(mu/find-first{:something :string
:request-m
[:map
[:item-a :something]
[:misc [:vector string?]]]}
(fn [schema _ _]
(some keyword? (-> schema mu/keys))))#2023-08-1604:06steveb8nlooks like mu/find-first has an opts arg where you should provide the registry. try that?#2023-08-1615:35Joel(def my-registry {:something :string
:request-m
[:map
[:item-a :something]
[:misc [:vector string?]]]})
(defn map-keywords? [schema opts]
(mu/find-first schema
(fn [schema _ _]
(some keyword? (-> schema mu/keys)))
opts))
(map-keywords? [:schema {:registry my-registry} :request-m] {})
I found that this works, using the options didn’t seem to help (how would it know which kvto use?
However, I’m puzzled, if :request-m is :request/m or a qualified keyword ::request then this does not work
(map-keywords? [:schema {:registry {:something :string
::request
[:map
[:item-a :something]
[:misc [:vector string?]]]}} ::request] {})
#2023-08-1607:32radhikahey, I have been generating singular constant values from schemas using :gen/elements [x], which works, but feels less idiomatic than perhaps a hypothetical :gen/return x (corresponding to clojure.test.check.generators/return that I looked for initially). any reason :gen/return isn't already provided for by malli.generate?#2023-08-1608:04ikitommino reason, goal has been to use same names & features as test-check. If/as it has return, :gen/return sounds good. Would you like to do a PR of this?#2023-08-1608:04radhikasure, will do#2023-08-1612:01radhikaheh, I'm having trouble extending this test data map because I don't know the language:
{:korppu "koira"
:piilomaan "pikku aasi"
:muuli "mukkelis"}
any suggestions for the next couple of themed entries that I can add to test gen/return?
I think I need the same shape as the first two entries, like:
:key1 "word"
:key2 "word word"
#2023-08-1612:56ikitommi:rolling_on_the_floor_laughing: here’s some extras:
:pikimusta "kissa"
:aasi "isaskari"
source: https://yle.fi/aihe/artikkeli/2012/09/18/elakoon-piilomaan-pikku-aasi#2023-08-1614:11radhikawell, I raised an issue https://github.com/metosin/malli/pull/932 and took a pass at it with PR https://github.com/metosin/malli/pull/933 (and some variation of fictional character names 😅)
feedback welcome, my first time venturing into changing malli sources 🙂#2023-08-1608:20ikitommipost-fixed the malli CHANGELOG for https://github.com/metosin/malli/blob/master/CHANGELOG.md#0103-2023-03-18. There was an unmarked BREAKINGchange with mu/strip-extra-keys-transformer stripping extra keys from :maps.#2023-08-1622:09markbastianI’m using malli to generate some swagger docs and have a field marked as optional (here’s the subset of the schema):
[:map
[:qp_column_name {:optional true} [:maybe :string]]]
This is in a map that is a key in the parent calling map (i.e. it isn’t a top level key). When I generate my swagger.json file, I get this fragment:
"description": {
"type": "string",
"x-nullable": true
},
It does not contain the "required": *true* attribute. Is there a way to set this?#2023-08-1622:20p-himikMaybe I'm reading it wrong, but why would the key be marked as required if it's already marked as optional?#2023-08-1622:50markbastianAh, I mistyped. Should have said false. it seems that the “parameters” key has required true or false for all entries. My body param looks like this:
{
"in": "body",
"name": "body",
"description": "",
"required": true,
"schema": {
The schema key itself seems to not have any required keys at all. Is there a way to set required on those keys (either true or false)?#2023-08-1700:59David GIs there a particular reason why there's map-of but the vector equivalent is vector in the schemas?#2023-08-1708:04ikitommino good reason. (bad) names could be revisited if there will be 1.0.0 to clean up things.#2023-08-1708:05ikitommigood thing is that you can always already add these to the registry:
:vector-of (-collection-schema {:type :vector-of, :pred vector?, :empty []})
:set-of (-collection-schema {:type :set-of, :pred set?, :empty #{}, :in (fn [_ x] x)})
...#2023-08-1716:42respatializedGiven a value v and a schema registry with a bunch of named schemas, is there a way to efficiently get all the schemas that v conforms to?#2023-08-1808:50ikitommiefficiently - you should loop the registry and create a map of name -> m/validator of it. Validators are pure functions and can be cached.#2023-08-1808:51ikitommiSee https://github.com/metosin/malli/blob/master/src/malli/provider.cljc#L15-L16 for example#2023-08-1809:23ikitommimaybe therw could be a helper for this in malli.util?#2023-08-2000:39jasonjcknThis might interest you https://github.com/esuomi/muotti @UFTRLDZEW#2023-08-2000:42respatializedYes, it looks like much more fully realized version of what I once called "https://fabricate.site/background/finite-schema-machines" - thanks for the link!#2023-08-1805:32mx2000Is there any tool , given schemas, that give me an 'edn-editor'.
For example I am developing a videogame and the damage effect is a tuple for example [:damage [:phyiscal [5 10]]]
I want to build a GUI where I want to be able to select for example damage-type :physical or :magic and so on.
Or choose from the available :effects. Or for example give a :creature an item from list of items available.#2023-08-1807:46valtteriThere’s at least malli playground https://malli.io/
sources: https://github.com/metosin/malli.io#2023-08-1807:48valtteriNot exactly what you’re looking for but maybe a starting point#2023-08-1807:48mx2000I want to build a GUI from schema for editing entities/components/etc#2023-08-1807:49mx2000Where user does not have to remember data schema#2023-08-1810:25valtteriYep, AFAIK there’s no ready-to-use malli compliant form-builder yet. We’ve been thinking about integrating malli with something like https://formik.org/ but nothing’s done yet.#2023-08-1812:03mx2000I was looking more for a desktop gui to set edn values.#2023-08-1812:03mx2000But I am using now an internal gui library of libgdx, which anyway not many people use outside of libgdx.#2023-08-1812:33ikitommirelated:
• https://escherize.com/works/data-desk/
• https://github.com/dvingo/malli-react-hook-form#2023-08-1815:04pithylessHave you considered a malli encode/decoder to CSV or XLS (and just using a spreadsheet editor)?#2023-09-1000:09Chip@U05476190 Though he’s using standard schema (not malli), that’s what @U0E703ECU did and https://github.com/mlimotte/tryion/blob/63bcdebf4e6bfd3b87433e1c3ce0f5cde566578f/src/tryion/db/db.clj#L48 from a google sheet. It’s cool.#2023-09-1000:25ChipAnd @US1LTFF6D built a https://clojurians.slack.com/archives/CLDK6MFMK/p1693997147327869#2023-08-2318:50anderI'm trying to write a schema for a map where a few keys are optional. These are coming in through a form post through ring/reitit and coercion is working well. I can get it so that it converts empty strings to nil but I think it'd be nice to dissoc it from the map entirely. I have something like this so far:
(defn omit-blank [s]
(when-not (str/blank? s)
s))
(def user (m/schema [:map {:closed true}
[:user/email :string]
[:user/alias {:decode/string omit-blank
:optional true} [:maybe :string]]]))
I found a similar post here, but I'm not sure where to put the function that was mentioned as a solution:
https://clojurians.slack.com/archives/CLDK6MFMK/p1653309841392379#2023-08-2401:55steveb8nQ: for a function schema (e.g. mx/defn) is there a sensible way to use a value from the inputs in the output schema? e.g. a max-height arg where the output schema checks that none of the results exceed that height#2023-08-2405:44ikitommicurrently there is no way for this, but there could be (like spec has). design draft/suggestion welcome on this#2023-08-2405:48ikitommialso, don’t think the inputs allow other than raw :cat, should take any schema there to support:
(s/fdef ranged-rand
:args (s/and (s/cat :start int? :end int?)
#(< (:start %) (:end %)))
:ret int?
:fn (s/and #(>= (:ret %) (-> % :args :start))
#(< (:ret %) (-> % :args :end))))#2023-08-2405:49ikitommiMaybe discuss over an issue? support would be simple, but better get the design right from the start.#2023-08-2405:53steveb8nsounds good. I’ll log one now#2023-08-2405:59steveb8nhttps://github.com/metosin/malli/issues/936#2023-08-2406:00steveb8nI’ll comment after you have a chance to think/type….#2023-08-2620:44Garrett HopperIt seems like there's a bug when dealing with [:altn] schemas with only one subsequence inside a :* sequence schema
(Probably not super important, as there's not much reason to use :altn with a single subsequence, but I did run into it when programmatically generating a schema that uses :altn and potentially only has one subsequence.
(m/validate [:* [:altn [:a [:= :a]]]] [:a]) throw a Wrong number of args (5) passed to: clojure.lang.PersistentVector error#2023-08-2620:44Garrett Hopper;; Fails with (Wrong number of args (5) passed to: clojure.lang.PersistentVector)
(m/explain ;; (or m/validate)
[:* [:altn
[:a [:= :a]]]]
[:a])
;; Works
(m/parse
[:* [:altn
[:a [:= :a]]]]
[:a])
;; Works
(m/parse
[:* [:altn
[:a [:= :a]]
[:b [:= :b]]]]
[:a :b])
;; Works
(m/validate
[:* [:altn
[:a [:= :a]]
[:b [:= :b]]]]
[:a :b])
;; Works
(m/explain
[:* [:altn
[:a [:= :a]]
[:b [:= :b]]]]
[:a :c])#2023-08-2620:45Garrett HopperFull stacktrace
Wrong number of args (5) passed to: clojure.lang.PersistentVector
AFn.java: 429 clojure.lang.AFn/throwArity
AFn.java: 48 clojure.lang.AFn/invoke
regex.cljc: 286 malli.impl.regex$_STAR__validator$_STAR_p__37514/invoke
regex.cljc: 152 malli.impl.regex$cat_validator$fn__37390$fn__37391/invoke
regex.cljc: 556 malli.impl.regex$validator$fn__37692/invoke
core.cljc: 2124 malli.core$validate/invokeStatic
core.cljc: 2118 malli.core$validate/invoke
core.cljc: 2122 malli.core$validate/invokeStatic
core.cljc: 2118 malli.core$validate/invoke#2023-08-2620:45Garrett Hopper{:mvn/version "0.11.0"}#2023-08-2707:15ikitommiPlease write an issue.#2023-08-2720:59radhikahello, if I have defined a :multi schema with a couple of branches, how can I tell mg/generate to produce a value from specifically one of those branches? the default behavior seems to randomly select from all :multi branches :thinking_face:#2023-08-2722:26steveb8nIf you compose the multi from another schema you can generate it directly. Possible with or without a registry#2023-08-2723:24radhikathanks, yeah, splitting it up is roughly the workaround I went with—actually not by using :multi at all, but dispatching by multimethod.
the trouble with extracting sub-schemas into their own definitions and using them in :multi is that it duplicates the dispatch circuit, which added more boilerplate and more connections for maintainers to wire up when extending the :multi schema. it would be much neater to produce a single generator for the :multi schema and let it reuse its dispatch function for generating too#2023-08-3020:00ChipJust picked up malli yesterday (and Clojure a few weeks ago) so I’m a superdork.
I’m doing something stupid here. I want to make sure I don’t have rogue cognito-ids hanging off non-people.
(ns life.db.schema.validators
(:require
[malli.core :as m]
))
(def life-with-user-schema
[:map
[:life/uuid {:type :uuid, :optional true}]
[:system/cognito-id {:type :string, :optional true}]])
(def valid-entity
{:life/uuid (java.util.UUID/randomUUID)
:system/cognito-id "some-cognito-id"})
(def invalid-entity
{:system/cognito-id "some-cognito-id"})
(println (m/validate life-with-user-schema valid-entity)) ; Should return true
(println (m/validate life-with-user-schema invalid-entity)) ; Should return false
I get this error for both
; Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
; :malli.core/invalid-schema
I obviously don’t know what I’m doing. Suggestions?#2023-08-3020:16radhikayou might want to rewrite the map entry definitions like [:life/uuid {:optional true} :uuid] so that the last item describes the entry's value#2023-08-3020:16radhikaalso given both entries are declared :optional true, the invalid-entity might not be invalid#2023-08-3020:42ChipI needed both of those suggestions. Works now. Here are the corrections FWIW
(ns life.db.schema.validators
(:require
[malli.core :as m]
))
(def life-with-user-schema
[:map
[:life/uuid {:optional false} :uuid]
[:system/cognito-id {:optional true} :string]])
(def valid-entity
{:life/uuid (java.util.UUID/randomUUID)
:system/cognito-id "some-cognito-id"})
(def invalid-entity
{:system/cognito-id "some-cognito-id"})
(println (m/validate life-with-user-schema valid-entity)) ; Should return true
(println (m/validate life-with-user-schema invalid-entity))
Thanks again.#2023-08-3109:43Stig BrautasetWhile not related to your question, did you know that in Clojure 1.11 and up you can replace (java.util.UUID/randomUUID) with (random-uuid) ?#2023-09-0116:29kennyWhen using m/walk, is there a way to update ref schemas in place? 🧵#2023-09-0116:29kennyTake the following example.
(def example-schema
[:schema {:registry {"A" [:map [:a string?]]
"B" [:map [:b string?]]}}
[:multi {:dispatch :type}
[:a "A"]
[:b "B"]]])
The :multi schema references the schemas defined in the inline registry. Calling m/walk on this will only walk to "A" and "B".
(m/walk
example-schema
(m/schema-walker
(fn [schema]
(prn schema)
schema)))
"A"
"B"
[:multi {:dispatch :type} [:a "A"] [:b "B"]]
[:schema {:registry {"A" [:map [:a string?]], "B" [:map [:b string?]]}} [:multi {:dispatch :type} [:a "A"] [:b "B"]]]
=> [:schema {:registry {"A" [:map [:a string?]], "B" [:map [:b string?]]}} [:multi {:dispatch :type} [:a "A"] [:b "B"]]]
I could do something like this.
(m/walk
example-schema
(m/schema-walker
(fn [schema]
(if (m/-ref-schema? schema)
(mu/update-properties schema assoc :foo "bar")
schema))))
=>
[:schema
{:registry {"A" [:map [:a string?]], "B" [:map [:b string?]]}, :foo "bar"}
[:multi
{:dispatch :type}
[:a [:malli.core/schema {:foo "bar"} [:map [:a string?]]]]
[:b [:malli.core/schema {:foo "bar"} [:map [:b string?]]]]]]
The problem is it inlines the schemas in the registry. I would like to walk to the registry and update the schema for "A" in the registry, so the output would look something like this.
[:schema {:registry {"A" [:map {:foo "bar"} [:a string?]]
"B" [:map {:foo "bar"} [:b string?]]}}
[:multi {:dispatch :type}
[:a "A"]
[:b "B"]]]
Is this possible now?#2023-09-0207:23ikitommiMorning. There is an option to m/walk to walk over the refs, but currently it inlines the refs while doing the walk. There could/should be a way to reverse the inlining when returning from the walk, so it would effectively update-in-place over refs. You are not the first to ask this (ping @U8MJBRSR5), please write an issue of this.#2023-09-0207:26ikitommiThis relates to how the local registeries are handled. Currently registries are ibternally merged together, which loses information on where the registries were designed.#2023-09-0404:39escherizemalli mentioned in this blog: https://kevinlynagh.com/newsletter/2023_09_hardware_prototyping/#2023-09-0411:38ikitommithanks for sharing, was interesting!#2023-09-0415:24escherizeYeah I was reading it, thinking “Malli fits this usecase super well” then shocked to see it mentioned directly 😂#2023-09-0405:21Karol WójcikHow can I define a Datomic Ref as a malli schema?#2023-09-0505:45steveb8nI might help if you can be more specific. we use Datomic and Malli together#2023-09-0506:05Karol WójcikHmm, I’m building set of scrapers for the same domain, but I get the data from different sources. Every scraper implementation has to satisfy a certain protocol, so then I can transact the results adapted to the unified model.
I would like to provide a schema for this edge of the system to make sure I’m not transacting 💩.
However, I’m not sure how the ref should be defined here. I wanted to use something like:
[:some.diatomic/attribute [:or TFullEntity TDatomicRef]]
Is there some predicate in the datomic API, so that I can check if the attribute is indeed a ref? #2023-09-0506:11Karol WójcikIt seems ref is just a number. Thought it’s some sort of a Ref type that is shown as a number. I will define it as a number? then ;) #2023-09-0506:33steveb8nah ok. we use uuids for all entities with a standard key and always using idents for refs. that way it’s easy to create a schema because every ref is [k uuid]#2023-09-0506:33steveb8ni.e. we never transact a ref using the :db/id#2023-09-0506:34steveb8ndoes that work for you? having uuids is generally a good idea for datomic entities in case you ever want to export/import#2023-09-0506:35steveb8nand yes, :db/id is just a number#2023-09-0521:00Karol WójcikHmm, this will not work for me. I'm not having uuids, but composite tuples since I'm not in control of the source of the data and for composite tuples I need to resolve the entity-ids 😄#2023-09-0521:00Karol WójcikThank you so much for help @U0510KXTU!#2023-09-0506:42steveb8nQ: I have a schema (from a registry) and I need its name (keyword). I found I can use m/-ref to get this value but I wonder about using a non-public fn. Is there a public api for this operation?#2023-09-0610:35Yevgeni TsodikovHi,
Is there a built-in schema for Java list/collection validation?
Something like [:vector :map], but for java.util.Collection?#2023-09-0610:39Yevgeni TsodikovI know I can create my own type with:
(m/-collection-schema {:type :java-collection, :pred #(instance? java.util.Collection %)})
But looking for alternatives#2023-09-0613:19ikitommino, there is nothing built-in. Same applies for concrete classes like Long, Integer. I you end up with a set of new schemas for java types, please share here.#2023-09-0613:21ikitommiI have a draft for effective types of schemas, so you derive generators, providers, transformers from effective type, :java-collection is kinda :vector. without this, you need to mount each new type separately.#2023-09-0610:45opqdonutI made a thing: https://github.com/opqdonut/malli-edn-editor#2023-09-0611:24ChipThat’s cool#2023-09-0612:45opqdonutthanks 🙂#2023-09-0614:50Samuel Ludwignoob "what am I doing wrong?" style question regarding :fn constraints and generation (contents in thread to not pollute channel):#2023-09-0614:50Samuel LudwigI'm playing around, and trying to create a spec of what an entry in some kind of version log might look like, with a :fields-changed (set), :before (map), and :after (map) key. I want to enforce a constraint that the keywords in the :fields-changed set are a subset of the keys in the :before and :after maps.
My spec is as follows:
(mg/generate
[:and
{:gen/fmap (fn [{:keys [fields-changed]}]
{:fields-changed fields-changed
:after (zipmap fields-changed (repeatedly #(rand-int 100)))
:before (zipmap fields-changed (repeatedly #(rand-int 100)))})}
[:map
[:fields-changed [:set keyword?]]
[:before map?]
[:after map?]]
[:fn (fn [{:keys [fields-changed before after] :as m}]
(tap> m)
(and (set/subset? fields-changed (-> before keys set))
(set/subset? fields-changed (-> after keys set))))]])
I suspect my :gen/fmap function is incorrect, and to tell the truth I'm not completely comfortable with generator stuff in general. I'm currently getting couldn't satisfy such-that errors every other generate call or so (it works when there's no keys generated)#2023-09-0614:55Stig BrautasetTIL about :gen/fmap. We use that quite a lot in Spec at work, and me thinking it was missing from malli was one reason why I didn't think it would work for us -- so thank you 🙂#2023-09-0614:59Stig BrautasetI can't tell what's wrong though...#2023-09-0615:02Samuel Ludwigyea, im not sure, zipmap should be kosher to use with sets, so I don't think its that#2023-09-0615:18Samuel Ludwigmy suspicion is that it might be checking the :fn relation before the generation maybe? but that would be counter to my purpose of providing a custom generator (to side-step hard-to-generate things)#2023-09-0615:25Samuel Ludwigthat tap> reveals maps that are clearly all randomly generated, and not a result of the fmap call; for example
{:fields-changed
#{:A./*?3+bY-L
:mn/+--+!**
:!*m/!e95:T
:sL:_/r
...
:_r3?/MKsW
:_8/S?-0T6},
:before
{-34238756986650630276294N 4,
true \;,
#uuid "485d16b8-7d0c-4f91-9d47-285b2f7921a2" :JQoO-:W,
...
"Vb*$&Qk7T" "I;qPI)"},
:after {true :Hg, 6 2}}#2023-09-0617:07Samuel Ludwigdoes the generator only 'attach' to the schema if its a :gen/gen "property" (think I'm using the correct term there)?#2023-09-0617:26ikitommi@U0482NW9KL1 you should move the :gen/fmap to the :map schema instead, this way, it's done before the :fn#2023-09-0617:28Samuel Ludwigahhh, so for reference, like so:
(mg/generate
[:and
[:map
{:gen/fmap
(fn [{:keys [fields-changed]}]
{:fields-changed fields-changed
:after (zipmap fields-changed (repeatedly #(rand-int 100)))
:before (zipmap fields-changed (repeatedly #(rand-int 100)))})}
[:fields-changed [:set keyword?]]
[:before map?]
[:after map?]]
[:fn (fn [{:keys [fields-changed before after] :as m}]
(tap> m)
(and (set/subset? fields-changed (-> before keys set))
(set/subset? fields-changed (-> after keys set))))]])#2023-09-0617:29ikitommi(there could be an :gen/ignore or similar to omit checking the extra constraints witj :and, but there is no such thing ATM)#2023-09-0617:29ikitommidoes that work?#2023-09-0617:30Samuel Ludwigit does!, interesting, didn't know the order had mattered like that 😅#2023-09-0617:39ikitommi:and generates on first child and validates on whole schema, :gen/fmap there works on the value returned, e.g. “too late”#2023-09-0707:46robert-stuttafordwhat is the correct spec to use for a seq that could be a vector or a list or a set? the docs say "You can use :sequential for any homogeneous Clojure sequence", but that's not true:
(for [v [[1]
(list 1)
#{1}]]
[v (m/validate [:sequential any?] v)])
;; ([[1] true]
;; [(1) true]
;; [#{1} false])
use case: a public API that can accept both EDN and JSON data. EDN uses sets, JSON uses arrays, which marshall to vectors#2023-09-0708:30radhikaa set isn't sequential? in clojure although it is seqable (order is not guaranteed). you could make an :or schema with :sequential :set to capture both value forms.
but I'd also consider canonicalizing the API input value into a set before processing further.#2023-09-0710:30ikitommiDo you want to retain the original collection type or convert internally to.. sets?#2023-09-0710:32ikitommiI think json transformers decodes vectors to sets automatically#2023-09-0710:33ikitommialso, there is mt/collection-transformer to change any (supported) collection type to another#2023-09-0710:34ikitommiNot sure why :sequential doesn't allow lists :thinking_face:#2023-09-0710:35ikitommiWhat does: (m/coerce [:set :int] [1 2] mt/json-transformer) do? (not at computer)#2023-09-0711:37robert-stuttafordthanks folks#2023-09-0807:07robert-stuttaford@U055NJ5CC :sequential does allow lists 🙂#2023-09-0708:48Bingen Galartza IparragirreHello! Is this behavior expected?
dev> (m/encode [:and [:keyword] [:enum :one :two]] :one mt/string-transformer)
"one"
dev> (m/encode [:enum :one :two] :one mt/string-transformer)
:one
#2023-09-0712:12Bingen Galartza IparragirreOk, I will!#2023-09-0712:16ikitommiIt's already fixed in master 😎#2023-09-0712:16Bingen Galartza IparragirreWow, that was fast! thank you#2023-09-0712:28ikitommihttps://github.com/metosin/malli/pull/951#2023-09-0814:17Ferdinand Beyer~How can I use schemas from my custom registry in malli instrumentation?~
~I have a namespace with a ~
~I now want to annotate functions like this, and use ~
(defn my-func
{:malli/schema [:=> [:cat :custom-schema] :any]}
[val])
~I’ve tried to use ~
~The only thing that did work is:~
(defn my-func
{:malli/schema (malli.core/schema [:=> [:cat :custom-schema] :any] {:registry registry})}
[val])
~Is that the recommended way to do it?~#2023-09-0814:21Ferdinand BeyerI was hoping to keep my metadata as plain data, without needing to require malli.core.#2023-09-0814:50Ferdinand BeyerArgh. I found out that my custom start! that calls set-default-registry! is actually never called. It does work.
Sorry for the noise 😇#2023-09-1009:09fuadOut of curiosity: where/when do you call set-default-registry? I tried using the default registry lately but moved away from it when I hit some load ordering problems.#2023-09-1012:13Ferdinand BeyerIn development, I just call set-default-registry! followed by malli.dev/start!.
My custom registry is a composite-registry of default-schemas and mutable-registry. The mutable registry points to an atom where I assoc my schemas to. I first used default-registry instead of default-schemas, which does not work when using set-default-registry! because of infinite recursion.
My production code does not change the default-registry, but passes the custom registry using the :registry option#2023-09-0917:46Sahil DhanjuI have this error being printed in the repl.
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
:malli.core/invalid-schema
Is it possible to print more information about what the specific issue is? The docs suggest that's possible#2023-09-0918:01ikitommitry *e in repl to see the exception data, which gives more info#2023-09-0918:01ikitommiwill push this soon: https://clojurians.slack.com/files/U055NJ5CC/F05MFHYKBQR/malli-pretty.png#2023-09-0918:01Sahil Dhanju#error{:cause ":malli.core/invalid-schema",
:data {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema nil}},
:via [{:type clojure.lang.ExceptionInfo,
:message ":malli.core/invalid-schema",
:data {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema nil}},
#2023-09-0918:01Sahil DhanjuIs this saying the schema is nil?#2023-09-0918:02ikitommiyes, not a good explanation thou.#2023-09-0918:03ikitommiHopefully the line number in the stack trace shows the location. I'll try to improve the error messages#2023-09-1012:13dvingoWhat do you think about including this change to the error log message? - specifically for cljs where you don't have *e bound it helps immeasurably to see the schema shape that failed:
https://github.com/dvingo/malli/commit/53d74e05d82ef812ceab181011894c694accba9d
(defn- -lookup! [?schema f rec options]
(or (and f (f ?schema) ?schema)
(if-let [?schema (-lookup ?schema options)]
(cond-> ?schema rec (recur f rec options))
(-fail! (str ::invalid-schema " - " (pr-str (cond-> ?schema (schema? ?schema) -form))) {:schema ?schema}))))#2023-09-1013:26ikitommiWe had that in https://github.com/metosin/reitit/pull/576 (pr-str data), but really slow and not good setting for backend production. For development, this would be great.#2023-09-1013:27ikitommiWhat would be a good way to separate dev & prod?#2023-09-1013:28ikitommimaybe dev/start! could override Vars like m/schema and plug in pretty printing. That would work in clj, maybe in cljs too?#2023-09-1015:03dvingoah that's a good point - for cljs there is goog/DEBUG - but a general malli specific solution would make sense!
for clj I think overriding a Var is a simple solution. For cljs either the goog/DEBUG route would work or having a malli specific flag that is toggled in the cljs preload#2023-09-1012:59kokonuthi, I am new to malli coming from spec and it looks like coerce is what is equivalent to spec's assert. Is it correct?
Then I am wondering if coerce in malli can be turned on and off like spec does by (s/check-asserts true) .#2023-09-1013:12kokonutOtherwise, I am thinking of doing like
(comment
(require '[malli.core :as m])
(def mode (atom true))
(defn my-coerce [definition data]
(if @mode
(m/coerce definition data)
data))
(reset! mode true)
(my-coerce :int "42") ;; thrown
(reset! mode false)
(my-coerce :int "42") ;; "42"
)
though not sure if this is a good practice.#2023-09-1013:23Ben SlessCoercion needs a transformer#2023-09-1013:23Ben SlessThink about a date coming in as a string#2023-09-1013:23Ben SlessYou know it should be a date,you want a date,so you turn it into a date before you check if it's valid#2023-09-1013:30kokonutIt seems coerce is a larger concept than I thought. What I am trying to do is simply checking type in the middle of a function (so not instrumentation), so I am thinking coerce without transformer. The github readme says
Coercion can be applied without transformer, doing just validation:
(m/coerce :int 42)
; 42
(m/coerce :int "42")
; =throws=> :malli.core/invalid-input {:value "42", :schema :int, :explain {:schema :int, :value "42", :errors ({:path [], :in [], :schema :int, :value "42"})}}
#2023-09-1013:32kokonutBut the convention is we (almost) always use it with transformers?#2023-09-1013:51ikitommiI think m/assert would make sense. Currently working with a codebase which has both spec and malli, and lot's of spec asserts. So, need that too before can fully migrate to malli.#2023-09-1013:53ikitommiAnd, nil is nowadays a valid transformer, so you can just validate & throw with coerce. m/assert could use that.#2023-09-1013:53ikitommiCould you write an issue of that @U022N1DU7GQ?#2023-09-1013:54ikitommiand, welcome to malli! Curious to hear how do you perceive the differences#2023-09-1013:55kokonutYes, I was looking for something like m/assert and thought m/coerce would be it.#2023-09-1013:57kokonutI am still new to malli but feel malli have more powerful features and is very flexible (data-driven). I also learned that if I want to do registry malli also supports it (so I guess it covers what spec does?)#2023-09-1013:59kokonutThe bigger question I had is how much/often I should use namespaced keywords in maps but usually I find I personally prefer plain keywords as they just look clean to me. With flexible validation tools like malli, I think I can go with plain keywords for lots of cases.#2023-09-1014:55Ben SlessI think you want validate in that case, not coerce#2023-09-1019:06jasonjckn@U022N1DU7GQ (s/check-asserts true) do you turn it on or off at runtime, or just set it at startup#2023-09-1019:07jasonjckns/check-asserts uses a volatile! for on/off switch, but i'm a fan of compile-time switches#2023-09-1019:09kokonutI usually turn it on on development and testing phase. On production I turn it off.#2023-09-1019:10jasonjckngreat, so don't need a volatile! for that#2023-09-1019:10jasonjckni have ~30 minutes before I have to leave but i'll try to send a malli PR before then#2023-09-1019:31jasonjckn#2023-09-1019:43jasonjcknPR https://github.com/metosin/malli/pull/953 (CC @U055NJ5CC)
https://github.com/jasonjckn/malli/tree/assert
alright, bbl 🙂#2023-09-1118:45Samuel Ludwighow should I write a m/=> function schema for a fn that takes in an open-ended option-map (where I need to ensure types if certain keys are provided)? for example:
(def Entity
(m/schema
[:map
[:id uuid?]
[:attributes map?]]))
(m/=> create-new [:=> [:cat ___?] Entity])
(defn create-new [& {:as kvs}]
(merge
{:id (random-uuid)
:attributes {}}
kvs))
which might be called as
(create-new
:attributes {:one 2 :three [4 5]}
:six 7)
not seeing any examples in the docs afaict (but I may have missed them)#2023-09-1221:54dvingoI'm not sure of a way to describe that either, but maybe a small addition to malli.core/-instrument can help:
In my fork I added the ability to do this:
(def Entity (m/schema [:map [:id uuid?] [:attributes map?]]))
(def Args (m/schema [:map
[:id {:optional true} uuid?]
[:other {:optional true} :string]
[:attributes {:optional true} map?]]))
(defn create-new
{:malli/schema [:=> [:cat Args] Entity]
:malli/xform-args (fn [& x] [(apply hash-map x)])}
[& {:as kvs}]
(merge
{:id (random-uuid)
:attributes {}}
kvs))
;; call with
(d/pre (pr-str (create-new :id "(random-uuid)")
https://github.com/dvingo/malli/blob/5e1ce89dbfa9a6fa55724e72a8aeffcf0479cbca/src/malli/core.cljc#L2564
which gives this error in cljs:#2023-09-1221:54dvingoThis was needed to get integration with the helix library where the function will receive a js object but by using a macro it appears to be a hashmap. So I needed a way to transform the arguments so malli would validate it correctly#2023-09-1222:06dvingonot sure if this would be accepted upstream, but you can copy dev/start! to use this version of instrument in your project#2023-09-1222:07dvingomaybe => can be updated to accept optional data as well so that version would support this transform#2023-09-1611:28ikitommitwo things in this thread:
1. there is no inbuilt schema to support kw-args, but you can describe them with :alt, example below. Issue welcome if you want a :kwargs utility schema for this
2. xform-args - looks interesting, is this just for validating or also as a “transform args behind the scenes” thing?#2023-09-1611:28ikitommimalli.desctructure has examples on how to describe the kwargs: https://github.com/metosin/malli/blob/master/test/malli/destructure_test.cljc#L123-L147#2023-09-1611:31ikitommiso, here it would be:
[:altn
[:map [:map
[:id uuid?]
[:attributes map?]]]
[:args [:*
[:alt
[:cat [:= :id] uuid?]
[:cat [:= :attributes] map?]]]]]#2023-09-1611:31ikitommior just:
[:alt
[:map
[:id uuid?]
[:attributes map?]]
[:*
[:alt
[:cat [:= :id] uuid?]
[:cat [:= :attributes] map?]]]]
#2023-09-1611:32ikitommibut, there could be something like:
[:sequential-map
[:id uuid?]
[:attributes map?]]
… that would behave like it.#2023-09-1612:04ikitommiand because it’s all data, this should work too:
(defn kwargs [m]
[:altn
["map" (m/form m)]
["args" [:* (into [:alt] (reduce (fn [acc [k _ v]] (conj acc [:cat [:= k] v])) [] (m/children m)))]]])
(kwargs
[:map
[:id uuid?]
[:attributes map?]])
;[:altn
; ["map"
; [:map
; [:id uuid?]
; [:attributes map?]]]
; ["args"
; [:*
; [:alt
; [:cat [:= :id] uuid?]
; [:cat [:= :attributes] map?]]]]]
#2023-09-1901:44dvingothanks for the pointers Tommi! good to know about :alt and :altn
For the xform-args - it's intended to only be used for validations, the function will get the original args unprocessed
It came from a use-case where helix, the react library, emits a function for you via a macro that will transform a JS object to hashmap, but I wanted to add instrumentation to by describing the args with a :map schema. This xform-args was one solution to get it working without needing to add more macro layers.
But it seems like it could be useful in other cases where a transformation can simplify the arguments schema validation#2023-09-1612:04ikitommiand because it’s all data, this should work too:
(defn kwargs [m]
[:altn
["map" (m/form m)]
["args" [:* (into [:alt] (reduce (fn [acc [k _ v]] (conj acc [:cat [:= k] v])) [] (m/children m)))]]])
(kwargs
[:map
[:id uuid?]
[:attributes map?]])
;[:altn
; ["map"
; [:map
; [:id uuid?]
; [:attributes map?]]]
; ["args"
; [:*
; [:alt
; [:cat [:= :id] uuid?]
; [:cat [:= :attributes] map?]]]]]
#2023-09-1421:08fabraoHello, how can we handle something like
(->
(malli.core/schema [:map
[:user [:map
[:name {:optional true} :string]
[:description {:optional true} :string]]]])
(malli.core/explain (json/decode "{\"user\": null}" true))
(malli.error/humanize)) => {:user ["invalid type"]}
if both are optional?#2023-09-1421:43Samuel Ludwigwhat do you want to be the result of this decoding? an error? or
{:user {}}?#2023-09-1421:44fabraowell, not result error#2023-09-1501:44radhikaif the whole value could be nil, you may want a https://github.com/metosin/malli/#maybe-schemas, like [:maybe [:map [:name …] [:description …]]#2023-09-1519:13Stuart NathHello Everyone,
I'm new to Malli and am trying to use it for referential integrity on my data. The following approach works, but is slow compared to other approaches, such as a http://tech.ml join.
The dataset is a vector of maps, where each map represents a row. There are 170k rows.
I am trying to conduct a referential integrity check on the "Item" key, of which there are 2,100 unique "Items".
I have followed the following process:
(def enum-statement (into [:enum] set-of-items))
(def m-schema [:map ["Item" enum-statement] ["ShipmentDate" :some] ["Tons" :double]])
(time (frequencies (pmap #(m/validate m-schema %) data))) ;; => 40 seconds
(time (frequencies (map #(m/validate m-schema %) data))) ;; => 90 seconds
Is there another way in Malli to accomplish this faster? If not, is there a way to make the current approach faster?
I still might end up using this because it is pretty elegant, but wanted to ask here to see if there was an easy performance boost available.#2023-09-1519:29Stuart NathUsing the validator function we got the time down to 1.7 seconds:
(time
(let [valid? (m/validator m-schema)]
(frequencies (pmap #(valid? %) demand-data))))
This has solved our speed issue. We will keep experimenting with it.#2023-09-1611:07ikitommiYes, m/validator returns a pure and (mostly) optimized function for this. I would assume map would be faster here, if the data is already read into memory. e.g. (map valid? demand-data)#2023-09-1813:31Stuart NathThat did turn out to be the case - map was 3x faster.#2023-09-1523:39Eli PinkertonHello all,
I'm new to Malli (as well as Clojure!) and am using it via gungnir (https://kwrooijen.github.io/gungnir/model.html) to define my db models. This is very cool. However, I have a problem - the data source for the objects that I want to put into my db don't always have matching field names. For example, the db colum for a table might be named "foo", but the actual JSON that I want to shove into that table's field is named "foobar". I want to map "foobar" to "foo", and potentially run some conversion functions on it (to change type from an int to a string, for example).
It looks like Malli supports transformers (https://github.com/metosin/malli#value-transformation), but I'm a bit confused as to how to use them. Would appreciate some guidance here!#2023-09-1912:26cheewahbelow works
(m/validate [:schema {:registry {:my.ns.enum/Enum [:enum :ZERO :ONE]
:my.ns.singular/Singular [:map [:enum_val [:ref :my.ns.enum/Enum]]]}}
[:ref :my.ns.singular/Singular]]
{:enum_val :ZERO})
; true
but if i put the schema into a registry (for re-use), it produces the following error
(def registry [:schema {:registry {:my.ns.enum/Enum [:enum :ZERO :ONE]
:my.ns.singular/Singular [:map [:enum_val [:ref :my.ns.enum/Enum]]]}}])
(m/validate [:ref :my.ns.singular/Singular] {:enum_val :ZERO} {:registry registry})
; Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
; :malli.core/invalid-ref
appreciate some pointers on how to do this correctly#2023-09-1918:14zy C(m/validate [:ref :my.ns.singular/Singular] {:enum_val :ZERO}
{:registry (malli.registry/composite-registry
m/default-registry
{:my.ns.enum/Enum [:enum :ZERO :ONE]
:my.ns.singular/Singular [:map [:enum_val [:ref :my.ns.enum/Enum]]]})})
try this?#2023-09-2001:07cheewahthis works, thanks for the help!#2023-09-2117:38Noah BogartIs there a way to annotate/validate def forms? I have an (assert (m/validate XSchema some-obj)) as a top-level form but that's ugly and doesn't actually check when evaluating the def later (like in the repl)#2023-09-2120:06escherizeThat would be handy to have.#2023-09-2204:32ikitommihttps://github.com/metosin/malli/pull/776#2023-09-2213:01Noah Bogartexcellent, thank you#2023-09-2200:32David GI'm getting some weird errors with my schema with the decoder/transformers and :+ and would love if someone can help me figure this out (commented inside the code block):
(require '[malli.core :as m]
'[malli.transform :as mt])
;; Malli doesn't seem to have a type that can be decoded for bigdec hence this more elaborate schema
(def bigdec-s (malli/-simple-schema
{:type :core/bigdec
:pred (partial instance? BigDecimal)
:type-properties
{:decode/json (fn [x]
(try (bigdec x)
(catch Exception _ x)))}}))
;; :+ doesn't work
(m/decode [:+ [:and bigdec-s pos?]]
["0"]
(mt/transformer
(mt/key-transformer {:decode keyword})
mt/json-transformer))
;; => ["0"]
;; :vector does work
(m/decode [:vector {:min 1} [:and bigdec-s pos?]]
["0"]
(mt/transformer
(mt/key-transformer {:decode keyword})
mt/json-transformer))
;; => [0M]
What's concerning is that m/explain just throws an exception (also does m/coerce):
(m/explain [:+ [:and bigdec-s pos?]]
(m/decode [:+ [:and bigdec-s pos?]]
["0"]
(mt/transformer
(mt/key-transformer {:decode keyword})
mt/json-transformer)))
;; => Execution error (ClassCastException) at malli.core/-simple-schema$reify$reify$explain (core.cljc:658).
;; class java.lang.String cannot be cast to class java.lang.Number (java.lang.String and java.lang.Number are in module java.base of loader 'bootstrap')
(m/explain [:vector {:min 1} [:and bigdec-s pos?]]
(m/decode [:vector {:min 1} [:and bigdec-s pos?]]
["0"]
(mt/transformer
(mt/key-transformer {:decode keyword})
mt/json-transformer)))
;; => {:schema [:vector [:and :core/bigdec pos?]],
;; :value [0M],
;; :errors ({:path [0 1], :in [0], :schema pos?, :value 0M})}
I can paste this into the Github repo as an issue if that's preferable#2023-09-2218:06David Ghttps://github.com/metosin/malli/issues/958#2023-09-2206:28steveb8n@ikitommi do you know of anyone using https://github.com/metosin/malli/blob/master/docs/clojurescript-function-instrumentation.md in a node server instead of browser?#2023-09-2206:29steveb8ninstrumentation is checking value but the errors thrown don’t include/display the explanation#2023-09-2206:29steveb8nI guess it’s either not possible or need a specific setup#2023-09-2206:30steveb8nI can workaround by capturing input values in REPL and using m/explain but the jvm DX is so nice I’d love to have it in the node env as well#2023-09-2211:43ikitomminot using in node, so just all 👂s with this one.#2023-09-2215:38dvingoI just tried it out with a small :node-script target and just need to change the default report argument from thrower to reporter:
https://github.com/metosin/malli/blob/2a5fc9a6bbf1360881104df2974dc319d3d17431/src/malli/dev/cljs.cljc#L29
in shadow-cljs.edn:
:instrument-node {:target :node-script
:output-to "out/instrument_node.js"
:main malli.instrument-node/main }
the ns:
(ns malli.instrument-node
(:require
[malli.core :as m]
[malli.dev.pretty]
[malli.dev.cljs :as dev]))
(defn plus [x] (inc x))
(m/=> plus [:=> [:cat :int] [:int {:max 6}]])
(defn minus
{:malli/schema [:=> [:cat :int] [:int {:min 6}]]}
[x] (dec x))
(defn main [& args]
(println "in main"))
(dev/start! {:report (malli.dev.pretty/reporter) :skip-instrumented? true})
(comment
(m/function-schemas :cljs)
(plus "hi")
(minus 5)
)
in the terminal where node is running you'll get printed output now instead of exceptions e.g.:
-- Schema Error ------------------------------------------------------------------------------------
Invalid function return value:
4
Function Var:
malli.instrument-node/minus
Function arguments:
[5]
Output Schema:
[:int {:min 6}]
Errors:
{:in [], :message "should be at least 6", :path [], :schema [:int {:min 6}], :value 4}
More information:
----------------------------------------------------------------------------------------------------#2023-09-2215:40dvingonot sure about "instrumentation is checking value but the errors thrown don’t include/display the explanation" I'm seeing even with the thrower I get the same report in the repl output it just doesn't show in the terminal#2023-09-2223:59steveb8nthanks @U051V5LLP you are right. I need to apologise because I just realised it was my own logging config that was not displaying the explanation in the node env#2023-09-2300:00steveb8nit is working as you demonstrated for me too#2023-09-2300:15steveb8nthat said, I have uncovered a challenge in this scenario. I have a try/catch block that is catching the exception thrown by malli.core/-fail#2023-09-2300:17steveb8nin the catch block I’m trying to log the explain. it’s not in the exception data. I’ve tried using m/explain using the :schema and :inputs data from the exception but it returns nil. can’t figure out why#2023-09-2300:31steveb8nhere’s a repro….#2023-09-2300:31steveb8n(mx/defn ^:malli/always combine :- :string
[{:keys [prefix suffix]} :- [:map
[:prefix :int]
[:suffix :string]]]
(str prefix " -> " suffix))
(md/start!)
(try
(combine {:prefix "1" :suffix "a"})
(catch :default e
(let [{:keys [args schema]} (:data (ex-data e))]
(malli.core/explain schema (first args)))))#2023-09-2300:31steveb8nthe explain in the catch block returns nil in the node env#2023-09-2300:32steveb8n@ikitommi @U051V5LLP any suggestions on this? I now suspect either mx/defn or ^:malli/always as the problem because they were necessary to reproduce this#2023-09-2311:47dvingono worries @U0510KXTU was going to say if you have a repro.., so thanks : ) yea I think the issue would be in mx/defn implementation. From a quick scan for any differences vs what dev/start! is doing it looks like the mx/defn does not pass the :report option to -instrument
https://github.com/metosin/malli/blob/30a2176f1893602d8a948ef3103fde91ec5fd638/src/malli/experimental.cljc#L56C39-L57C84
and those reporters are what add the :data and :type https://github.com/metosin/malli/blob/30a2176f1893602d8a948ef3103fde91ec5fd638/src/malli/dev/pretty.cljc#L88 to the ex-info map
so could try adding that to the mx/defn implementation#2023-09-2311:49dvingoalso looks like both code paths in mx/defn (with and without malli/always) don't add :report#2023-09-2408:08steveb8nBrilliant. I might try fixing this. Will PR if I do#2023-09-2623:11steveb8n@ikitommi I just tried to fix mx/defn but I get lost in the connections between mx/defn calling -instrument and how that data flows to pretty. could you give me a nudge in the right direction about what needs to change for pretty to have the data it needs from mx/defn? I’ll test the fix locally once it is working and will PR#2023-09-2623:33steveb8ntbh this shows my lack of macro-foo. I tried requiring pretty into the experimental ns and then creating a pretty/thrower to use in the macro body. but fails at runtime because I haven’t correctly spliced in the thrower fn.#2023-09-2623:35steveb8ngpt4 tells me that cljs macros can’t use fns because they aren’t constants. at this point I’ll be guessing so I’ll wait for some cljs macro Jedi master instruction#2023-09-2213:42markbastianIs there a way to specify a protocol using malli? Something like this:
(defprotocol MyProtocol)
(mc/validate MyProtocol (reify MyProtocol)) ;:malli.core/invalid-schema
#2023-09-2214:06skynetdon't know if there's a better way, but you could do
[:fn #(satisfies? MyProtocol %)]
#2023-09-2214:08markbastianCool! Works for me. Thanks!#2023-09-2214:53Craig BrozefskyI asked a similiar question a few months ago, this is what I have in a common library now:
(defn protocol-schema
"Returns a malli schema that ensures the value satisfies the given protocol."
[prot]
(m/-simple-schema
{:type (:on-interface prot)
:pred #(satisfies? prot %)
:type-properties {:error/fn (fn [error _] (str "Expected value to satisfy " (:on-interface prot) " protocol."))}}))#2023-09-2214:54Craig BrozefskyIt MAY be worth considering what happens with protocols being redefined, and change that to take a symbol that is resolved in the pred....#2023-09-2214:55Craig BrozefskyI saw may because I have not confirmed my hunch that it makes my schema a little delicate in the face of protocol redefs#2023-09-2215:01Noah BogartWe have a validation middleware set up in our ring app that has historically used json-schema, and i'm introducing malli to it. everything works for the most part, but I'm running into slight awkwardness. I have a POST endpoint that expects a payload with a string uuid, and in the compojure handler I'm decoding the provided json to clojure/edn. When the endpoint validation was using json-schema, this worked nicely because json-schema expects a string and would verify that it could be parsed to uuid and then the malli decoding transformed the validated string to uuid. and I could use the same malli schema for both instrumentation and for validation because m.js/transform converted my :uuid malli schema to a {"type": "string", "format": "uuid'} json schema.
Now I'd like to use the same :uuid schema for both payload validation and instrumentation. is my best bet just to write a (def payload-coercer (m/coercer PayloadSchema (mt/transformer mt/json-transformer))) and use that in the payload instead of using the schema itself? has anyone else run into this situation?#2023-09-2518:51ikitommiI'm not sure I understand what is the question here.#2023-09-2518:51Noah Bogartlol i'm sorry#2023-09-2518:52Noah Bogarti think the question is, what's the best way to have a given malli schema represent both input to validate and internal model?#2023-09-2518:52Noah Bogartis that possible (or desirable)?#2023-09-2518:59ikitommisure, m/coerce , m/coercer , m/assert or just function schemas:
(mx/defn no-op :- PayloadSchema [p :- PayloadSchema] p)
#2023-09-2519:00ikitommido you want just dev-time validation or also runtime validation?#2023-09-2714:34Noah Bogartboth. i wired up a ring middleware which calls m/validate on a provided malli schema. i could call m/coerce, but that's not cached and i'm hoping to avoid writing manual (def ObjCoercer (m/coercer ObjSchema)) for every schema.#2023-09-2714:34Noah Bogartmaybe i could just write a coercer cache myself#2023-09-2715:06ikitommiwait what, which routing library are you using? Reitit at least has built-in mw for request and response coercion with cached coercer.#2023-09-2715:07Noah BogartWe're using Compojure and we have a homespun url-specs system that's very reminiscent of Reitit but predates it by a while#2023-09-2715:08Noah Bogarti've suggested moving to reitit, but we have roughly 350 endpoints and switching is hard lol#2023-09-2715:09ikitommiCached now resolution in compojure is really hard, things like context bodies are resolved for each request at runtime#2023-09-2715:09ikitommicompojure or compojure-api?#2023-09-2715:16Noah Bogartjust compojure#2023-09-2715:19Noah Bogartwe're not caching anything in the compojure routes at all, we have a separate map that is filled with entries like this:
[:get "/v0.1/public-invite-codes/:public-invite-code"] {:response #'invites/public-get-response}
and when resolving the request, the validation middleware calls (get url-specs (:compojure/route request)) and then for each of the keys in the spec, it calls the related validation function#2023-09-2715:20Noah Bogart(when payload-schema (validate-payload route payload-schema payload)) etc#2023-09-2715:20Noah Bogartit's pretty simple but does the trick#2023-09-2417:04skynetI think there's an issue with native-image in the new malli 0.13.0:
...
[2/8] Performing analysis... [] (5.8s @ 0.67GB)
6,974 reachable types (83.6% of 8,338 total)
6,895 reachable fields (43.0% of 16,053 total)
25,327 reachable methods (49.1% of 51,556 total)
1,354 types, 87 fields, and 770 methods registered for reflection
Fatal error: com.oracle.svm.core.util.VMError$HostedError: Invalid Class.forName value for malli.core.IntoSchema: interface malli.core.IntoSchema
at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:78)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:137)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.VMError.guarantee(VMError.java:156)
at org.graalvm.nativeimage.builder/com.oracle.svm.core.hub.ClassForNameSupport.registerClass(ClassForNameSupport.java:59)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.reflect.ReflectionDataBuilder.registerTypesForGenericSignature(ReflectionDataBuilder.java:726)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.reflect.ReflectionDataBuilder.registerTypesForGenericSignature(ReflectionDataBuilder.java:683)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.reflect.ReflectionDataBuilder.registerTypesForGenericSignature(ReflectionDataBuilder.java:677)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.reflect.ReflectionDataBuilder.registerTypesForClass(ReflectionDataBuilder.java:564)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.reflect.ReflectionDataBuilder.registerHeapDynamicHub(ReflectionDataBuilder.java:968)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.heap.SVMImageHeapScanner.onObjectReachable(SVMImageHeapScanner.java:165)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.heap.ImageHeapScanner.lambda$markReachable$5(ImageHeapScanner.java:452)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.heap.ImageHeapScanner.lambda$postTask$14(ImageHeapScanner.java:695)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
...#2023-09-2418:19lreadOthers are hitting this symptom too... see thread: https://clojurians.slack.com/archives/C02FBBU61A9/p1695550217143999?thread_ts=1695455634.060119&cid=C02FBBU61A9#2023-09-2418:25lreadhttps://github.com/oracle/graal/issues/7486#2023-09-2518:07Samuel LudwigWhats the prescribed manner in which to write => function specs for defmulti forms/clauses, where the signature varies between each defmethod? Not seeing anything specific in the docs#2023-09-2518:55ikitommithere is nothing for that at the moment, but would :multi get you anywhere close to what you want?#2023-09-2519:00Samuel Ludwigthats what i thought of first, but i was unsure "where" the :multi defs belonged in the schema/how i would dispatch#2023-09-2519:05Samuel LudwigAs in, suppose I wrapped multiple :=>'s in the :multi, and I was dispatching based off the first argument of the multi-fn: how should i reference that first argument?#2023-09-2519:38Samuel Ludwigseparate question, is there a go-to way to denote an atom, or a delayed value? also can't find a clear answer on the docs#2023-09-2904:32ikitomminothing bult-in, but you can use :fn or m/-simple-schema to create such behavior#2023-09-2904:32ikitommie.g. [:fn delay?].#2023-09-2720:35shaderhow do I spec a map that is open homogenous except for a few declared keys that can have different types?#2023-09-2721:26shaderah, I think I'm looking for the 'default' idea#2023-09-2904:30ikitommi@shader yes, ::m/default key allows you to mixin the two:
(m/validate
[:map
["key" :string]
["value" :int]
[::m/default [:map-of :keyword :boolean]]]
{"key" "123"
"value" 123
:extra true})
; => true#2023-09-3005:10timothypratleyHi 🙂
Given the schema:
[:schema
{:registry
{"hiccup" [:orn
[:kindly [:fn {:error/message "should have kindly metadata"} kind]]
[:primitive [:or string? number? boolean? nil?]]
[:fragment [:or
[:and seq?
[:catn [:children [:* [:schema [:ref "hiccup"]]]]]]
[:and vector?
[:catn [:fragment-indicator [:= :<>]]
[:children [:* [:schema [:ref "hiccup"]]]]]]]]
[:tag-node [:and vector?
[:catn [:tag simple-keyword?]
[:attrs [:? [:and [:not [:fn kind]]
[:schema [:ref "attrs"]]]]]
[:children [:* [:schema [:ref "hiccup"]]]]]]]
[:raw-node [:and vector?
[:catn [:raw-indicator [:= :hiccup/raw-html]]
[:content string?]]]]
[:component-node [:and vector?
[:catn [:view-fn [:and fn?
[:function [:=> [:cat any?]
[:schema [:ref "hiccup"]]]]]]
[:children [:* any?]]]]]
[:reagent-node [:and vector?
[:catn [:component [:or [:and list? [:cat [:= 'fn]]]
symbol?]]
[:args [:* any?]]]]]
[:scittle-node [:and vector?
[:catn [:forms [:+ list?]]]]]]
;; TODO: attrs may be over restricted (why not accept anything string-able?) and under specified (only styles can be maps)
"attrs" [:map-of
[:or string? keyword? symbol?]
[:or string? keyword? symbol? number? boolean? nil? vector?
[:schema [:ref "attrs"]]]]}}
"hiccup"]
Trying to explain a very basic failure:
(html [:div [:h2 "hi"] '(info)])
me/humanize returns:
["should have kindly metadata" "should be a string" "should be a number" "should be a boolean" "should be nil" "should be a seq"]
Which doesn't point to the actual error: '(info) is not valid hiccup.
Any suggestions for how to get a more informative error message in this scenario?#2023-09-3007:17ikitommiyou can call malli.error/with-error-messages on the explain result to access all information. Does that help?#2023-09-3007:18ikitommibut, for sequence schemas, the errors are not always very nice, might report on (intuitively) wrong branch on :alt|n.#2023-09-3007:20ikitommibut, if the explain (with error messages) has all the info you see important here, you can plug custom resolver via :resolve option to reduce the most relevant error.#2023-09-3007:21ikitommihope this helps#2023-09-3007:22ikitommialso, suggestions to make the default error resolving in humanize better most welcome!#2023-09-3016:35timothypratleywith-error-messages creates something so long I can't paste it in this chat 😞
What I'd like to see is either something like this:
[:div [:h2 "hi"] *ERROR]*
or
[:div [:h2 "hi"] '(info)]
^ERROR
because the outer div and inner h2 are fine, it is just the form at the end that is bad.
Maybe this is very difficult because the grammar might say oh but the entire expression is not hiccup because of the error. From a human point of view, the outer expression is fine, but because there is an error at the tail position, it invalidate the entire expression. I don't know how to create a (recursive) grammar that can eagerly accept that some parts are correct, only a small part is incorrect.
One idea that comes to mind is that I could allow '(info) to be considered correct in the grammar (match any? ) and then handle that clause as an error in my code. That solves the problem for coding, but then the grammar is no longer useful for sharing as a standard.
Have you ever encountered such a situation?#2023-10-0402:30hifumi123What would be the best way to access properties of keys in a map schema? Suppose we have
(def example
[:map
[:foo {:default :x} [:enum :x :y :z]]
[:bar {:default 0} :int]])
The best I could think of is:
(reduce (fn [result [key properties]]
(assoc result key properties))
{}
(m/children example))
But if a built-in malli utility exists for this, it would prefer that.#2023-10-0712:42CaseyI spent about 15 minutes trying to implement field level error messages for top level fn map errors then I noticed that already exists! https://github.com/metosin/malli#custom-error-messages
What a brilliant feature! Makes validating forms where the rules for an attr changes based on other attrs really easy! ❤️ malli#2023-10-0807:00DrLjótssonWhat is the easiest way to make all top-level keys in a schema optional?
[:map
[:a [:map [:b :string]]]
[:c [:map [:d :int]]]]
;; =>
[:map
[:a {:optional true} [:map [:b :string]]]
[:c {:optional true} [:map [:d :int]]]]
#2023-10-0807:27jasonjcknhttps://github.com/metosin/malli/blob/0.13.0/src/malli/util.cljc#L232#2023-10-0808:46DrLjótssonThanks! And now I see that it's even mentioned in the docs...#2023-10-0913:31CaseyIs there any existing tools for consuming openapi schemas and producing malli schemas from the openapi data models?#2023-10-0923:16hifumi123if i recall correctly, OpenAPI uses JSON schema, so you would want to try the patches in https://github.com/metosin/malli/pull/211#2023-10-1007:05CaseyAh that's the search keyword I was missing. I'll have a crack at that patch.#2023-10-0919:45sergey.shvetsHi, I'm trying to validate that one and only one of two keys in a map is present. e.g {:block/prop1 1 :block/prop2 2 :block/children []} or {:block/prop1 1 :block/prop2 2 :block/markup "markup"} I want to make sure that only markup or children key is present. Is there a better way to do it without going for multi-schema? Ideally, I'd like to validate keys separately and then for keys that present validate that their attributes/values are of a proper type/shape.#2023-10-1005:59opqdonutyou could use [:and [:map ...] [:fn ...]], see https://github.com/metosin/malli#fn-schemas#2023-10-1014:01dvingoI was playing around with an idea of adding a :custom-pred property to the -map-schema
https://app.slack.com/client/T03RZGPFR/CLDK6MFMK
https://github.com/dvingo/malli/commit/1f3d00f8abbc0aafede329d558931f110427f3b5
That would support use-cases like yours in an extensible way.
Something like:
[custom-map-schema {:exclusive-keys #{:block/markup :block/children}}
[..etc]]
I think it would be similar to the :compile property supported by -collection-schema but I'm not exactly sure how that works
https://github.com/metosin/malli/blob/39ccfef96b54beb3d862b1eab5f5be90ec0f4456/src/malli/core.cljc#L1188#2023-10-1104:04sergey.shvetsThanks!#2023-10-1014:13cheewahHi, am wondering if it is possible to define custom validation function that takes in optional params, and then refer to it somewhere else? Something like :vfunc below.
(def schema {:registry (mr/composite-registry
m/default-registry
{:vfunc [:fn (fn [v & opts] true)] ; opts = {:min 10}
:./msg [:and [:map {:closed true}
[:int32_val {:optional true} int?]]
[:vfunc {:min 10}]]})}) ; use vfunc and pass in {:min 10} as optional argument
(m/validate [:ref :./msg] {:int32_val 1} schema)
(above code doesn't work as it is invalid schema, but it expresses what I am trying to achieve)#2023-10-1015:33escherizeIs there a third party library or other similar thing for putting malli schemas on defprotocols, defrecords, and/or multimethods?#2023-10-1017:14escherize(or am I going to be the one to do it?) 😉#2023-10-1018:09escherizegranted, multimethods aren’t actually required to return the same shape, but neither are functions and having malli-ized defns (and fns) is pretty nice.#2023-10-1022:35Stig BrautasetIf Malli supported schemas on protocols I'd have a strong case for adopting it at work. We have quite a lot of code to work around the inability to put Specs on protocols.#2023-10-1101:11dvingohttps://clojurians.slack.com/archives/CLDK6MFMK/p1665585081681289
I haven't heard anything since this ^#2023-10-1101:11dvingohttps://blog.ambrosebs.com/2022/09/08/schema-defprotocol.html#2023-10-1101:12dvingohttps://github.com/plumatic/schema/pull/432#2023-10-1106:05hifumi123IIRC pre/post conditions are intentionally not supported in protocol methods. The workaround is to make an instrumented function wrapping the protocol method. i.e. given -f you define f with pre/post conditions#2023-10-1116:15escherize@U051V5LLP thanks for posting that, very helpful#2023-10-1021:48steveb8nQ: when using nested schemas from a registry, is there a way to recursively deref all the way down? I can see something like this in mu/subschemas but I just want everything local. this would make transformations much easier#2023-10-1101:16ambrosebsmalli.core/deref-all ?#2023-10-1110:35steveb8nUnfortunately doesn't recurse multiple levels#2023-10-1111:48eval2020This?
(let [schema [:schema {:registry {"More" [:map
[:more boolean?]]
"Other" [:map
[:other boolean?]
[:more "More"]]}}
[:map
[:this boolean?]
[:that "Other"]]]]
(m/walk schema (m/schema-walker m/deref-all)
{::m/walk-schema-refs true ::m/walk-refs true}))
;; => [:map [:this boolean?] [:that [:map [:other boolean?] [:more [:map [:more boolean?]]]]]]#2023-10-1122:38steveb8n@U04V6FEES thank you!! that is exactly what I needed#2023-10-1122:38steveb8nfeels like that should be a util fn or in docs somewhere. very useful when working with deep registries#2023-10-1214:39dvingocould be good to add here
https://github.com/metosin/malli/blob/master/docs/tips.md#2023-10-1221:42steveb8nexcellent suggestion. I’ll pay forward and will submit a PR for it#2023-10-1401:41steveb8nhttps://github.com/metosin/malli/pull/967#2023-10-1109:55Stig Brautaset👋 I found that the Malli transformer does not support transforming from json strings to :uri . Is this because ClojureScript lacks uri? support? Would a PR that adds :uri support for Clojure only stand any chance of being accepted?#2023-10-1111:00opqdonutYeah it's a good starting point, at least. We also have clj-only for ratio?.#2023-10-1112:10Stig BrautasetSomething like this? https://github.com/metosin/malli/pull/966#2023-10-1117:39isakAuthorization is normally done outside of malli, correct? E.g., making sure someone owns the db id they want to update.#2023-10-1120:10Stig BrautasetI'm not an authority on this but that is my take, yes.#2023-10-1120:42isakAm I wrong?
(defmulti authorize! (fn [auth-key auth-values context] auth-key))
(defmethod authorize! :update-project [_ auth-values context]
(next.jdbc/execute (:db-conn context) ...))
(let [context {:user-id 1 :db-conn ...} ;; from outside scope
!authorization-required (volatile! (transient #{}))
update-projects-schema [:vector
[:map
{:closed true}
[:id [:and
:int
[:fn (fn [id]
(vswap! !authorization-required conj! [:update-project id])
true)]]]
[:title :string]]]]
(when-not (m/validate update-projects-schema [{:id 1 :title "foo"} {:id 2 :title "bar"}])
(throw (ex-info "Validation failed" {:explain ...})))
(let [authorization-required (-> !authorization-required deref persistent!)]
(doseq [[auth-kind auth-values] (group-by first authorization-required)]
(authorize! auth-kind (mapv second auth-values) context))))#2023-10-1207:53hifumi123i would avoid doing this mostly because, for performance, i like to pre-compile validator and explainer functions for schemas of API data. the two big problem for me with this approach is namely:
• i expect validators to be pure functions, i.e. its safe to run the same validator function multiple times in different parts of my code
• if I am writing a schema for API data, I like being able to re-use the data across backend and frontend. by adding backend-specific code into a schema, portability is killed (and in your example, it seems that validators are side-effecting, which is rare for a function that takes in data and returns a boolean value)#2023-10-1207:54hifumi123in general, i like to use malli for validating and coercing the shape of data. in my opinion, business logic should not be complected into code that normally runs “at the boundaries” between functions#2023-10-1214:20isakYea that is fair. Though I see the same thing could be done with a malli transformer, which seems like it takes care of those issues (compilation, performance, schema reuse). So it would be a transformer that just collects some data but leaves values as-is. See any issues with that approach?#2023-10-1215:02isakActually it looks like transformers are supposed to be pre-built, so the performance concern remains, unless the values are gathered into a ^:dynamic var. Seems like malli isn't really intended to support a context, I'll go with another approach.#2023-10-1216:44escherizeI could use some help with something where mu/update-in seems to be misbehaving. Am I holding it wrong or is it broken? 🧵#2023-10-1216:45escherizeUltimately I’m trying to add a branch to an :orn schema, and not seeing what I expected:
Given
(def my-schema
[:schema {:registry {"test" [:orn [:string-node :string]]}} "test"])
(mu/update-in my-schema [0 0] mu/assoc :int-node :int)
;; => [:schema {:registry {"test" [:orn [:string-node :string]]}} "test"]
I expected to see an [:int-node :int] path under the :orn.
Not sure why the above doesn’t work when below seems to do the right thing
(mu/get-in my-schema [0 0])
;; => [:orn [:string-node :string]]
(mu/assoc [:orn [:string-node :string]] :int-node :int)
;; => [:orn [:string-node :string] [:int-node :int]]
#2023-10-1301:00steveb8nI’ve been doing a lot of this lately. maybe I can help#2023-10-1301:01steveb8nany kind of transform using registries is tricky. it’s much easier to deref the entire schema before starting any transforms#2023-10-1301:01steveb8nhttps://clojurians.slack.com/archives/CLDK6MFMK/p1696974522814249#2023-10-1301:02steveb8nonce you have it fully local, then transforms are much easier to grok. does this help?#2023-10-1402:43escherizeI ended up using m/form and a postwalk. I have control over the shape of the schema coming in, so it’s really just a superstition sort of thing.#2023-10-1402:44escherizeit is for a convenience function for adding branches to my hiccup->html compiler https://github.com/escherize/huff#extendable-grammar 🙂#2023-10-2219:10ikitommiSo, all good with this one?#2023-10-1218:50DrLjótssonWhat is the difference between the :tuple and :cat schemas?#2023-10-2219:04ikitommi:tuple is a fast fixed size vector. :cat is sequential schema with regex semantics. More powerful, bit slower.#2023-10-2219:07ikitommithe perf difference between the two:
(require '[criterium.core :as cc])
(let [valid? (m/validator [:* int?])]
(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 2.7µs
(let [valid? (m/validator [:sequential int?])]
(cc/quick-bench (valid? (range 10)))) ; Execution time mean : 0.12µs
#2023-10-2405:30DrLjótssonThanks!#2023-10-1719:04Brett RowberryI can’t get a custom error message to appear in an :or schema. Is this expected or a bug?
(me/humanize
(malli/explain
[:or
{:error/message "hi"}
:string
:int]
1.1))
expected:
["hi"]
actual:
["should be a string"
"should be an integer"]#2023-10-1719:05Brett RowberryWhat I’m really interested in is a schema for a string exactly 6 or 10 characters in length.#2023-10-1719:27pithylessI thought this would do it, but it still returns a separate entry for each :or child:
(-> [:or
{:error/message "hi"}
[:string {:min 2 :max 2}]
[:string {:min 4 :max 4}]]
(malli/explain "aaa")
(malli-error/humanize
{:resolve malli-error/-resolve-root-error}))
;; => ["hi" "hi"]#2023-10-1719:29Brett RowberryThis met my needs
(def 6-or-10-chars
[:re
{:error/message "Must be 6 or 10 characters"}
#"(^\w{6,6}$|^\w{10,10}$)"])#2023-10-1719:29Brett RowberryBut it still seems like :or should respect :error/message.#2023-10-1803:18hifumi123@brett.rowberry Let’s start with the following example.
(def schema
[:or {:error/message "1"}
[:string {:error/message "2"}]
[:int {:error/message "3"}]])
(m/explain schema nil)
;; {:schema [:or #:error{:message "1"}
;; [:string #:error{:message "2"}]
;; [:int #:error{:message "3"}]]
;; :value nil
;; :errors ({:path [0]
;; :in []
;; :schema [:string #:error{:message "2"}]
;; :value nil}
;; {:path [1]
;; :in []
;; :schema [:int #:error{:message "3"}]
;; :value nil})}
As you can see, we get errors for each alternative that failed. Here is the explainer function for an or-schema
(-explainer [_ path]
(let [explainers (-vmap (fn [[i c]] (-explainer c (conj path i))) (map-indexed vector children))]
(fn explain [x in acc]
(reduce
(fn [acc' explainer]
(let [acc'' (explainer x in acc')]
(if (identical? acc' acc'') (reduced acc) acc'')))
acc explainers))))
So to “support” a single error message for or-schema, we will want to add a check inside the explain function like so
(fn explain [x in acc]
(if-let [overall-error (:error/message (-properties this))]
(conj acc (miu/-error [] in this x))
(reduce (fn [acc' explainer] ...) acc explainers)))
and now we get the desired result:
(mr/set-default-registry! (merge (m/default-schemas)
{:modified-or (m/-modified-or-schema)}))
(-> [:modified-or {:error/message "blah"} :string :int]
(m/explain nil)
(me/humanize))
;; => ["blah"]
(-> [:modified-or :string :int]
(m/explain nil)
(me/humanize))
;; => ["should be a string" "should be an integer"]
---
@ikitommi Is this a patch worth sending? I am still not sure if it makes sense for an or-schema to have a canonical choice of error message.#2023-10-1806:09pithylessFor what it's worth, I came across a similar issue this week with :or where I expected it would allow me to override it with an overall :error/message or :error/fn . It came as a surprise, but I just assumed I'm doing something slightly wrong. I never dug into it further, since I ended up solving the problem differently.#2023-10-1806:10pithylessBTW, I had guessed maybe I need to wrap it - something ala [:schema {... custom-error-paths? ..} [:or ...]] - so I'm not saying it needs to be directly supported by :or, but it would be nice if there was a way to override the many-children-error results#2023-10-1816:42mpoffaldI would also benefit from this. I’ve been working on making our validation errors more readable to our users, and the many-children-error results have proven to be a real sticking point.
If I could add canonical messages to our :or s, it would be much more readable for our users than a vector of child-errors#2023-10-2219:02ikitommiHi. I’m in for doing a fix for this. Some thoughts:
• spec also reports errors on all or branches:
(require '[clojure.spec.alpha :as s])
(s/explain-data (s/or :int int? :string string?) :invalid)
;#:clojure.spec.alpha{:problems ({:path [:int], :pred clojure.core/int?, :val :invalid, :via [], :in []}
; {:path [:string], :pred clojure.core/string?, :val :invalid, :via [], :in []}),
; :spec #object[clojure.spec.alpha$or_spec_impl$reify__2112
; 0x741cc739
; "clojure.spec.alpha$or_spec_impl$reify__2112@741cc739"],
; :value :invalid}
• the output of resolution of root error is unfortunate:
(me/humanize
(m/explain
[:or {:error/message "hi"}
:string
:int]
1.1)
{:resolve me/-resolve-root-error})
; => ["hi" "hi"]
• … but I think it’s the cleanest way to handle this, e.g. schema just emits explanations fast, user can post-process this as sees fit. resolving the root error is a generic function, but with :or it creates duplicates.
• … we could just drop duplicates in m/humanize? with that:
(me/humanize
(m/explain
[:or {:error/message "hi"}
:string
:int]
1.1)
{:resolve me/-resolve-root-error})
; => ["hi"]
• would that be ok?#2023-10-2305:35ikitommialso, maybe the -resolve-root-error should be default resolve :thinking_face:#2023-10-2402:46hifumi123The behavior I would personally expect from malli’s or-schema is the following
• no root-level error -> report the error of each leaf
• root-level error -> report the root-level error, ignoring the leaves
this should not penalize performance too much IMO, so I am not sure if -resolve-root-error is a better default than malli’s current default
With that said, I am just a single user of malli. User research should help telling us what kind of API people prefer#2023-10-2407:28pithyless> • … we could just drop duplicates in m/humanize? with that:
m/humanize already does things like aggregating errors, etc. so I guess this would be a working solution. I don't understand enough of the details to make any judgement on alternative implementations.
> also, maybe the -resolve-root-error should be default resolve
I only discovered this due to the OP, so at the very least making this more discoverable and explaining the differences would be appreciated.
Any proposed solution would need to work for [:or :orn :alt :altn] - correct? And for both :error/message and :error/fn#2023-10-1803:18hifumi123@brett.rowberry Let’s start with the following example.
(def schema
[:or {:error/message "1"}
[:string {:error/message "2"}]
[:int {:error/message "3"}]])
(m/explain schema nil)
;; {:schema [:or #:error{:message "1"}
;; [:string #:error{:message "2"}]
;; [:int #:error{:message "3"}]]
;; :value nil
;; :errors ({:path [0]
;; :in []
;; :schema [:string #:error{:message "2"}]
;; :value nil}
;; {:path [1]
;; :in []
;; :schema [:int #:error{:message "3"}]
;; :value nil})}
As you can see, we get errors for each alternative that failed. Here is the explainer function for an or-schema
(-explainer [_ path]
(let [explainers (-vmap (fn [[i c]] (-explainer c (conj path i))) (map-indexed vector children))]
(fn explain [x in acc]
(reduce
(fn [acc' explainer]
(let [acc'' (explainer x in acc')]
(if (identical? acc' acc'') (reduced acc) acc'')))
acc explainers))))
So to “support” a single error message for or-schema, we will want to add a check inside the explain function like so
(fn explain [x in acc]
(if-let [overall-error (:error/message (-properties this))]
(conj acc (miu/-error [] in this x))
(reduce (fn [acc' explainer] ...) acc explainers)))
and now we get the desired result:
(mr/set-default-registry! (merge (m/default-schemas)
{:modified-or (m/-modified-or-schema)}))
(-> [:modified-or {:error/message "blah"} :string :int]
(m/explain nil)
(me/humanize))
;; => ["blah"]
(-> [:modified-or :string :int]
(m/explain nil)
(me/humanize))
;; => ["should be a string" "should be an integer"]
---
@ikitommi Is this a patch worth sending? I am still not sure if it makes sense for an or-schema to have a canonical choice of error message.#2023-10-2219:02ikitommiHi. I’m in for doing a fix for this. Some thoughts:
• spec also reports errors on all or branches:
(require '[clojure.spec.alpha :as s])
(s/explain-data (s/or :int int? :string string?) :invalid)
;#:clojure.spec.alpha{:problems ({:path [:int], :pred clojure.core/int?, :val :invalid, :via [], :in []}
; {:path [:string], :pred clojure.core/string?, :val :invalid, :via [], :in []}),
; :spec #object[clojure.spec.alpha$or_spec_impl$reify__2112
; 0x741cc739
; "clojure.spec.alpha$or_spec_impl$reify__2112@741cc739"],
; :value :invalid}
• the output of resolution of root error is unfortunate:
(me/humanize
(m/explain
[:or {:error/message "hi"}
:string
:int]
1.1)
{:resolve me/-resolve-root-error})
; => ["hi" "hi"]
• … but I think it’s the cleanest way to handle this, e.g. schema just emits explanations fast, user can post-process this as sees fit. resolving the root error is a generic function, but with :or it creates duplicates.
• … we could just drop duplicates in m/humanize? with that:
(me/humanize
(m/explain
[:or {:error/message "hi"}
:string
:int]
1.1)
{:resolve me/-resolve-root-error})
; => ["hi"]
• would that be ok?#2023-10-2112:21roklenarcicI get these kind of spec failures:
:in [0 375 :roklenarcic.security/underlying :roklenarcic.security/category],
:message "missing required key",
:path [0 0 :roklenarcic.security/underlying 0 :roklenarcic.security/category],
What’s the difference between :in and :path . In this case the actual element that is wrong is at 375, so why is path index [0 0 ….?#2023-10-2218:45ikitommihi. Added documentation to this: https://github.com/metosin/malli/commit/142fefd3978fd0fbd0991cde48cbc3641a2ab33d#2023-10-2218:46ikitommie.g. :path is a path in Schema, not value.#2023-10-2312:32roklenarcicok thanks#2023-10-2500:06mpoffaldIf :in is the path in the value, does that make this a bug in path->in?
(require '[malli.core :as m ])
(def my-schema
(m/schema [:orn
[:a-branch [:map [:a :int]]]
[:b-branch [:map [:b :int]]]]))
(let [{:keys [errors schema]} (m/explain my-schema {:a "2"})]
(mu/path->in schema (:path (first errors))))
;;=>
[:a-branch :a] #2023-10-2500:07mpoffaldmy value doesn’t have an :a-branch key#2023-10-2505:53ikitommiOh, it’s a bug.#2023-10-2505:55ikitommi@UNCHWPV4Y could you file an issue out of this? should be a simple fix. I think same applies to :altn and :catn.#2023-10-2604:43ikitommilooked at this with the morning ☕: https://github.com/metosin/malli/pull/969#2023-10-2500:06mpoffaldIf :in is the path in the value, does that make this a bug in path->in?
(require '[malli.core :as m ])
(def my-schema
(m/schema [:orn
[:a-branch [:map [:a :int]]]
[:b-branch [:map [:b :int]]]]))
(let [{:keys [errors schema]} (m/explain my-schema {:a "2"})]
(mu/path->in schema (:path (first errors))))
;;=>
[:a-branch :a] #2023-10-2214:59bzmarianoHello folks, I have the following error message :test-chuck-not-available, and I can't find any information about it.
This error occurs when evaluating (malli.generator/sample user-schema), where user-schema is:
[:map {:closed true}
[:username [:string {:min 8 :max 32}]]
[:password [:string {:min 16 :max 32}]]
[:email [:re email-regex]]
[:role [:enum "admin" "user"]]]
If I remove the email vector , the sample function works fine so the problem must be in the :re tag#2023-10-2215:37delaguardoyou have to add com.gfredericks/test.chuck "0.2.10" as a dependency for your project.#2023-10-2215:38delaguardohttps://github.com/metosin/malli#value-generation
that is mentioned here#2023-10-2215:45bzmarianooh I missed that comment line, thank you.#2023-10-2318:53hanDerPederRunning into an issue with map entry attributes in the map schema.
I haven't studied the implementation here so maybe I'm in undefined behaviour land, but since this works:
(mg/generate
[:map {:registry {:foo/bar :string}}
:foo/bar]) ;; #:foo{:bar "mWG3BP81X49181eod38930Y122oV"}
and this works
(mg/generate
[:map {:registry {:foo/bar :string}}
[:foo/bar]]) ;; #:foo{:bar "7IuHp3NFf"}
I would expect this to work:
;; expecting this to work
(mg/generate
[:map {:registry {:foo/bar :string}}
[:foo/bar {:min 100 :max 101}]]) ;; #:foo{:bar "7pU1jsi6NEnBa30MBp"} <-- too short :(
works if I write it out as documented:
(mg/generate
[:map {:registry {:foo/bar :string}}
[:foo/bar [:foo/bar {:min 100 :max 101}]]]) ;; #:foo{:bar "q348W......"} <-- works
Is this a bug, undefined behaviour or just not implementing?#2023-10-2410:09pithylessIIUC, in the third example, you are applying options to the key, not the value of the key-val association:
[:map [:foo/bar {:min 100 :max 101}]]
;; i.e.
[:map [:foo/bar {:min 100 :max 101} :foo/bar]]
Malli already has some amount of convenience (ie. magic) with how it tries to interpret these things, e.g:
[:map [:foo/bar]]
;; i.e.
[:map [:foo/bar :foo/bar]]
I don't see how Malli could figure out you want the third version to apply to the value and not to the key (e.g. to set {:optional true})#2023-10-2410:13pithyless(malli/ast
[:map {:registry {:foo/bar :string}}
[:foo/bar {:min 100 :max 101}]])
;; => {:type :map,
;; :keys
;; #:foo{:bar
;; {:order 0,
;; :value {:type :malli.core/schema, :value :foo/bar},
;; :properties {:min 100, :max 101}}},
;; :registry #:foo{:bar {:type :string}}}
^ yep, property of key not value#2023-10-2516:04hanDerPederHaven't used ast before, thanks!#2023-10-2609:34danielnealHow do you add properties to the key of a map-of schema? I’d like to specify that it’s a map of a path to a map-of an http-method to a value.
Not sure where to put the props
[:map-of {:title "path"} ;;; #2023-10-2610:34steveb8n#2023-10-2614:14ikitommi#2023-10-2614:23danielneal#2023-10-2811:09Ramon RiosFolks, Can I, based in one value, define what should be the schema for other property?
Example:
{:key1 2
:other-key ""}
{:key1 3
:other-key "should not be empty"}
[:map
[:key1 number?]
[:other-key (if (= 2 :key1 )
empty?
other-validation)]]#2023-10-2817:06ikitommiMaybe :multi ?
[:multi {:dispatch :key1}
[2 [:map [:other-key :string]]]
[3 [:map [:other-key :int]]]]#2023-10-2811:25raymcdermottwondering if anyone has a port of https://github.com/clojure/core.specs.alpha for #malli? Asking for a friend 😇#2023-10-2818:40hifumi123even if there were, it wouldn't be useful since the clojure compiler only let's clojure.spec validate macros#2023-10-2908:39raymcdermottIt can still be useful for linters, analysers and editors.#2023-10-3018:38Abhijit KumbharHi
I am new to malli.
Could someone please help me in defining schema for below sample payload- here "some-dynamic-value" is dynamic string value
{
"some-dynamic-value": {
"test": 0.07041698796202,
"test2":1}
}#2023-10-3018:47skynetyou could try
[:map-of {:size 1} :string [:map ["test" :double] ["test2" :int]]]#2023-10-3018:49Abhijit KumbharThank you @U0DSU64Q1 - I will try this.#2023-10-3019:12Abhijit Kumbhar@U0DSU64Q1 - I tried this , I am not getting any error but getting 200 OK with empty response {}#2023-10-3018:50Abhijit KumbharAlso need help in defining schema for dynamic payload to same endpoint, my endpoint accepting two types of payload below, how can I define final schema?
(def payload1
[:map
[:test [:vector :int]]
[:abc :string]
[:params
[:map
[:one :string]
[:two :string]
[:three :string]
]]])
(def payload2
[:map
[:test [:vector :int]]
[:abc :string]
[:params
[:map
[:testing :string]
[:start_month :string]
[:gender :string]
]]])#2023-10-3018:51skynetuse [:or payload1 payload2]#2023-10-3018:53Abhijit Kumbharlike this?
(def final
(malli.core/schema
[:or payload1 payload2])
)
#2023-11-0123:14Stig BrautasetI've managed to do this:
toy.malli> (def Foo
[:map
[:foo-bar {:decode/json csk/->kebab-case
:encode/json csk/->snake_case}
keyword?]])
#'toy.malli/Foo
toy.malli> (m/encode Foo {:foo-bar :foo-bar-baz} mt/json-transformer)
{:foo-bar "foo_bar_baz"}
toy.malli> (m/decode Foo {:foo-bar "foo_bar_baz"} mt/json-transformer)
{:foo-bar :foo-bar-baz}
I love that I can do this transformation on the values, but what I really want is to do that transformation on the keys.
I am using Malli to spec a public API, and I want the keys to use snake_case in the JSON over the wire, but kebab-case in Clojure. Is this possible?#2023-11-0123:15Stig BrautasetE.g. what I want is something more like:
toy.malli> (m/encode Foo {:foo-bar :foo-bar-baz} mt/json-transformer)
{:foo_bar :foo-bar-baz}
toy.malli> (m/decode Foo {:foo_bar :foo-bar-baz} mt/json-transformer)
{:foo-bar :foo-bar-baz}#2023-11-0123:35hifumi123I would probably run (de)munge function as I send (and receive) data. I want to say a custom transformer will do the job#2023-11-1600:06Ryan Barbercheck out the key-transformer that's already part of the transformer namespace: https://github.com/metosin/malli/blob/master/test/malli/transform_test.cljc#L548 -#2023-11-0908:18AkizHi, how would you describe a hash-map that can be empty or has the exact structure?
This [:or [:map [:id :uuid] [:name string?] [:age integer?]] [:map]] doesn’t work well with transformers.#2023-11-0909:09Martín VarelaIf you want the map to be empty, you could make it closed:
(m/validate [:map {:closed true}] {})
;; => true
(m/validate [:map {:closed true}] {:a :b})
;; => false#2023-11-0909:10Martín VarelaIf you don't, it will just validate any map, not imposing the need for it to be empty#2023-11-0909:11Martín VarelaNot sure about the transformers part, though.#2023-11-0913:44AkizSorry, I left it out because It doesn’t make any difference.
(malli.core/encode [:or [:map {:closed true} [:id :uuid] [:name :string] [:age :int]] [:map {:closed true}]] {:id (random-uuid) :name "john" :age 15} malli.transform/json-transformer)
=>
{:id #uuid "e707c673-61c3-42f6-b5ad-94b6d42dfe73", :name "john", :age 15}
and
(malli.core/encode [:map {:closed true} [:id :uuid] [:name :string] [:age :int]] {:id (random-uuid) :name "john" :age 15} malli.transform/json-transformer)
=>
{:id "d9073882-8310-426f-85c9-c8263569b37f", :name "john", :age 15}
In second case, the UUID get’s transformed#2023-11-0913:47Martín VarelaFor me it results in the same (json-encoded) value:
(malli.core/encode [:or
[:map {:closed true} [:id :uuid] [:name :string] [:age :int]]
[:map {:closed true}]]
{:id (random-uuid) :name "john" :age 15}
malli.transform/json-transformer)
;; => {:id "b1c74ece-f881-4a8f-8a19-a572a7a4d1bc", :name "john", :age 15}
(malli.core/encode [:map {:closed true} [:id :uuid] [:name :string] [:age :int]] {:id (random-uuid) :name "john" :age 15}
malli.transform/json-transformer)
;; => {:id "defafe57-e589-4e7b-9e57-ea9fe413f3bb", :name "john", :age 15}#2023-11-0913:48Martín Varela(modulo the UUID being different, of corse)#2023-11-0913:50Martín VarelaJust to check, which version of Malli are you running?#2023-11-0914:00Akiz0.11.0#2023-11-0914:02Akizi am on old version probably, thanks for head up. I will try to upgrade.#2023-11-0916:54Martín VarelaNot sure if it's that (I ran this on 0.12), but the exact code you had pasted worked for me. Maybe try upgrading, just in case#2023-11-1008:42AkizYeah, it was really the old version 🙂!#2023-11-1008:54Martín VarelaAwesome! Glad it's sorted 🙂#2023-11-0917:05hadilsHi! I am programming a GraphQL interface in Clojurescript. I have a recursive schema for my data structures in Malli, and I wanted to use multi-schemas as well, to select the correct starting point for a recursive schema. How would i go about doing this? I appreciate any comments or feedback offered.#2023-11-1020:22dvingoSomething like this?
(m/validate
(m/schema
[:schema {:registry {"top-level"
[:multi {:dispatch :type}
[:one
[:map
[:type [:= :one]]
[:one-prop :string]
[:children {:optional true} [:vector [:ref "top-level"]]]]]
[:two
[:map
[:type [:= :two]]
[:two-prop :int]
[:children {:optional true} [:vector [:ref "top-level"]]]
]]]}}
[:ref "top-level"]]
)
{:type :one :one-prop "hi" :children [{:type :two :two-prop 400}]})#2023-11-1323:03theequalizer73Hi folks, I’m using reitit.ring.coercion/coerce-request-middleware with malli, what can I do to avoid losing keys that are not defined in my schema. For example. If I send this parameters
{:parameters {:body {:username "my-username"
:email "
and my schema is something like
(def MySchema
(malli.core/schema [:map [:username :string]
[:email :string]]))
map schemas are open by default, but I’m not able to get the :phone parameter in my handler once it goes through the coerce-request-middleware unless I explicitly define it. I would like to be able to get any key that is not explicitly defined in my schema. What should I do?#2023-11-1405:38ambrosebsAre the extra keys dropped altogether, or just not being converted to keywords? Can you (get body "phone")?#2023-11-1405:40ambrosebsDepending on the answer to that, you might want to try something like:
(def MySchema
(malli.core/schema [:and [:map [:username :string]
[:email :string]]
[:map-of :keyword :any]]))#2023-11-1405:46ikitommi@U055XFK8V nowadays, you can define the “extra keys” like Schema does:
(m/validate
[:map
[:x :int]
[:y :int]
[::m/default [:map-of :int :int]]]
{:x 1, :y 2, 1 1, 2 2})
; => true#2023-11-1405:49ikitommibut, :map has also the :closed property:
1. [:map] - open by default (but reitit closes all maps for better web security)
2. [:map {:closed true}] - explicitly closed
3. [:map {:closed false}] - explicitly open
… with reitit, I would:
• if it’s just one schema, use the option 3
• if it’s all schemas, I would look for the malli coercion options, you can remove the “close by default” thing#2023-11-1418:23theequalizer73@U055XFK8V the extra keys are dropped altogether.#2023-11-1418:25theequalizer73Thanks @ikitommi will try those options.#2023-11-1422:07theequalizer73@ikitommi Are there more details about these two options in https://github.com/metosin/reitit/blob/620d0c271175a4e11d91d922b26c8162660db3f9/doc/coercion/malli_coercion.md#configuring-coercion?
;; add/set default values
:default-values true
;; malli options
:options nil
#2023-11-1422:08theequalizer73For the record, setting :strip-extra-keys false and mu/open-schema worked like a charm! Thx#2023-11-1514:53theequalizer73Hi @ikitommi You said “(but reitit closes all maps for better web security)“. What kind of risks are we facing if we open all schemas?#2023-11-1614:21Ben SlessStorage, injection, denial attacks, are all risks you take on when you let any data into your system at large#2023-11-1514:53theequalizer73Hi @ikitommi You said “(but reitit closes all maps for better web security)“. What kind of risks are we facing if we open all schemas?#2023-11-1410:27emilaasaIs anyone using malli to generate typescript definitions? Is there a simple example somewhere?
I looked at https://github.com/flowyourmoney/malli-ts but I was unable to understand it that well.#2023-11-1416:13valtteriHow complex cases are you aiming for? If simple + crude is OK, I’d start with malli -> json schema -> ts with existing tooling#2023-11-1509:15emilaasaNice suggestion valtteri, I'll try that!#2023-11-1509:37valtteriPlease let me know how it turns out. 🙂 I remember doing something similar as an exercise ~1.5 years ago#2023-11-1614:03theequalizer73Anyone? ^ 🙂#2023-11-1900:43FlavaDaveHello,
I would like to define a schema with namespaced keywords and then coerce a map that does not have namespaced keywords into a valid map with the appropriate namespaced keywords. Something like 👇
Can anyone help me come up with a good way to do this?
(ns super.cool.namespace
(:require
[malli.core :as mc]
[malli.error :as me]))
(def Schema
[:map
[::foo :string]
[::bar :into]])
(def data {:foo "clojure"
:bar 2})
(mc/coerce Schma data ?)
=> {:super.cool.namespace/foo "clojure"
super.cool.namespace/bar 2}#2023-11-1909:02ikitommiMaybe something like this?#2023-11-1909:03ikitommiI think it would be simpler if it just would create a transforming map in :compile
{:foo ::foo, :bar ::bar}
… so, no :ns declaration would be needed.#2023-11-1909:04ikitommimaybe this could be a built-in transformer in malli, with both decode & encode?#2023-11-1909:04ikitommiping @UF41YH1CM#2023-11-1922:36FlavaDaveThanks @U055NJ5CC that example worked for me.
I agreed that needing the :ns key isn't ideal. I think that having a map could be helpful if you want to do something articulate. But wouldn't it be best to just be able to get the namespace from the namespace of the key?#2023-11-2114:43Colin P. HillWhat is the difference between merge and union? I can see that the latter is implemented on top of the former, but the practical semantic difference is not clear to me.#2023-11-2119:27ambrosebsI brushed up on it myself and submitted a doc PR https://github.com/metosin/malli/pull/973#2023-11-2119:32Colin P. HillFabulous, that clears things up very nicely, thank you so much!#2023-11-2208:36ikitommimerged, thanks @U055XFK8V!#2023-11-2214:17Alex SkyHello! First of all, thank you for malli. It's beautiful.
So, my question is as follows
How properly show in swagger-ui structure like a :multi or :or
For example
[:multi {:dispatch :type}
[:sized [:map [:type keyword?] [:size int?]]]
[:human [:map [:type keyword?] [:name string?] [:address [:map [:country keyword?]]]]]]
(defmethod accept :multi [_ _ children _] (let [cs (mapv last children)] (assoc (first cs) :x-anyOf cs)))
As I understand it, here we take only the first part and the rest is not visible in swagger-ui.
i.e. in the model in swagger-ui I will see only :sized
I'd appreciate any ideas
or maybe is it ok for swagger?)
UPD someone is using swagger-ui ? and has used anyOf ?#2023-11-2409:31dergutemoritzI don't think :multi can be fully represented in swagger since the dispatch function can be any function. In our project we therefore introduced a custom :multi-map which specializes for a :type-key dispatch style multi schema whose children all have to be :map or :multi-map. This allows for representing it in less expressive formats (TypeScript declarations in our case - I don't know if swagger / open-api has support for this).#2023-11-2409:31dergutemoritzAs for :or I would assume that swagger should support this :thinking_face:#2023-11-2410:54Alex Skythanks for the reply. Yes it looks like there are no simple solutions 🙂#2023-11-2411:37ikitommiOpenAPI supports discriminator, which should help here if the :dispatch uses a key, see https://swagger.io/docs/specification/data-models/inheritance-and-polymorphism/#2023-11-2411:40ikitommimaybe we should support also functions when doing transformations to xyz from malli? so, instead of:
[:multi {:openapi {:type ....}}]
we could have:
[:multi {:openapi (fn [schema options] ...transform here...)}]
… that would allow easy in-place transformations.#2023-11-2411:40ikitommibut, PR welcome if the current :openapi doesn’t properly use the discriminator!#2023-11-2412:26dergutemoritzSpecal-casing keyword-based dispatch is a nice idea 👍#2023-11-2518:08Alex Skyoh https://github.com/swagger-api/swagger-ui/issues/2438
that's my problem.#2023-11-2520:09Alex SkyI guess I'm just on an older version.#2023-11-2520:11Alex Skymigrated to the new version now everything is fine. Problem solved.#2023-11-2518:55roklenarcicWould it be possible to precompile malli or something?
(time (require 'malli.dev))
"Elapsed time: 2972.572124 msecs"
It’s quite an expensive thing to load it.#2023-11-2519:16bocajFor loading in development time or for production use?#2023-11-2519:25Alex Sky(time (require '))
"Elapsed time: 302.174125 msecs"
:thinking_face:#2023-11-2519:26bocajWhat about using criterium?#2023-11-2519:28Alex Skyit's not about criterium or some benchmarks test
as I understand it. Probably a local problem?#2023-11-2519:32bocajYeah I’m assuming precompiling a dev namespace removes advantage of late binding while using the repl. If used in production would be compiled anyway. #2023-11-2607:21ikitommiI've been asking robust tools for AOTing libraries for years. There are ways to do this, but I have not found anything that is robust:
• https://clojure.org/guides/dev_startup_time
• https://clojureverse.org/t/deploying-aot-compiled-libraries/2545
Please share if you find a way to precompile malli that just works#2023-11-2607:26ikitommiuser=> (time (require '[malli.core]))
"Elapsed time: 409.494875 msecs"
#2023-11-2607:27ikitommiuser=> (time (require '[clojure.core.async]))
"Elapsed time: 1615.888209 msecs"
#2023-11-2607:27ikitommi🤷#2023-11-2607:27ikitommispec is precompiled by the core team, so:
user=> (time (require '[clojure.spec.alpha]))
"Elapsed time: 0.202042 msecs"
#2023-11-2616:50roklenarcicI don’t know what to tell you, takes almost 3 seconds on my Macbook pro 2019. Between Malli, tick with another 1.6sec load, clojure data xml with another 1.5sec load, it takes 10 sec to just load dependencies for my very simple command line utility.#2023-11-2616:50roklenarcicIt’s a real bummer#2023-11-2617:47ikitommiI hear you. You can always ask Clojure Core team for help here. For command line, I would recommend #CLX41ASCS, much snappier.#2023-11-2818:30FlavaDaveWent looking through Slack history and issues and it doesn't seem like there has been much talk about making Malli compatible with ClojureDart. I have recently been doing more with ClojureDart and I think that making Malli compatible with that runtime would be a huge improvement. Has anyone here made an attempt at that?#2023-11-2818:35ikitommiHaven't heard either, but it would be great to have that! (Don't even know what it mean to be ClojureDart-compatible)#2023-11-2818:36FlavaDave@U055NJ5CC I wont have any bandwidth for this for a few weeks but I could take a swing at it if you would like#2023-11-2818:40FlavaDavelooks like we would need to just throw some :cljd reader conditionals in a few places to make it work.#2023-11-2819:45Alex Skyas I know cljDart doesn't have multimethods yet and that could probably complicate things.#2023-11-2819:47FlavaDaveThat could indeed complicate things #2023-11-2906:17ikitommiok. so - core, errors, utils, inferring and transforming would work oob, but not the following:
• generators
• json-schema/swagger
• pretty printing#2023-11-2906:19ikitommigenerators also support protocols, but as clojure doesn’t have a good story for protocols implemented for protocols (which is IMO one of the core problems in Clojure itself), doesn’t work that either with current codebase.#2023-11-2906:19ikitommiideas welcome 🙂#2023-11-2908:09cgrand👋 Please come in touch with us before attempting the CLJD port. We have a couple of things to fix to make the experience pleasant. The biggest one is switching the reader from LispReader.java to contrib reader to have sane conditional behavior. (A port can be pulled off in the current state but it requires stupid and useless workaround — we learnt it the hard way by porting Datascript)#2023-11-2908:29baptiste-from-paris+1 with @U3E46Q1DG and we can help 🙂#2023-11-2913:38ikitommiAwesome, thanks guys 🙇#2023-11-2918:05FlavaDave@U3E46Q1DG @U2N9GDB1U do you think that multimethods would need to be implemented first as well?#2023-11-2909:15Marius(-> [:and [:map
[:password string?]
[:password2 string?]]
[:fn {:error/message "passwords don't match"
:error/path [:password2]}
(fn [{:keys [password password2]}]
(= password password2))]]
(m/explain {:password "secret"
:password2 "faarao"})
(me/humanize))
; {:password2 ["passwords don't match"]}
Hi all! In this example, it is possible to programmatically define :error/path depending on the validation function (and the data to be validated)?#2023-11-2913:51ikitommicurrently, no - but I have bumped into this too. Ideas for this:
1. :error/explain that takes value schema options as args and returns maybe an error-map with :path , :message etc. (like explain does
2. same, but new schema [:explain (fn [value schema _options] {:path …, :value …, …}]#2023-11-2913:51ikitommicomments welcome.#2023-11-2913:56Mariusok, thank you!#2023-11-2914:03ikitommicould you write an issue out of this, will forget otherwise#2023-11-2914:52Mariussure, will do#2023-11-3008:18MariusI wrote https://github.com/metosin/malli/issues/975
The description is hopefully good enough#2023-11-3008:09Patrick DelaneyI was reading through the malli docs, and tried the following:
user=> (require '[malli.provider :as mp] '[malli.transform :as mt])
nil
user=> (mp/provide
#_=> [{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"
#_=> :time "2021-01-01T00:00:00Z"}
#_=> {:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"
#_=> :time "2022-01-01T00:00:00Z"}]
#_=> {::mp/value-decoders {'string? {:uuid mt/-string->uuid
#_=> 'inst? mt/-string->date}}})
[:map [:id :string] [:time :string]]
but was expecting:
; => [:map [:id :uuid] [:time inst?]
is there another example of using malli.provider/value-decoders or did the conventions for using it change? I'm on 0.13.0#2023-11-3008:28ikitommihi. the docs are not up-to-date, could you PR a fix here? what has changed:
• provider creates real schemas instead of predicates, e.g. :string over string?
• this applies to the ::mp/value-decoders so, this is the new correct:
(mp/provide
[{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"
:time "2021-01-01T00:00:00Z"}
{:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"
:time "2022-01-01T00:00:00Z"}]
{::mp/value-decoders {:string {:uuid mt/-string->uuid
'inst? mt/-string->date}}})
; => [:map [:id :uuid] [:time inst?]]#2023-11-3008:46Patrick Delaneythank you for your help! I'll make a PR.#2023-11-3017:58twashingI’m getting an OutOfMemoryError when calling malli.core/validate... where the return type is a Malli SchemaObject.
Both of these calls with the below input. I did notice someone asking a https://clojurians.slack.com/archives/CLDK6MFMK/p1617265988295000?thread_ts=1617177945.267200&cid=CLDK6MFMK. But there doesn’t seem to be a resolution.
Is there something else I need to know about fn validation?
(m/explain =>fn-schema my-fn)
(m/validate =>fn-schema my-fn {:gen/max 1})
(def SchemaMap
[:map
[:foo/a string?]
[:foo/b inst?]])
(def SchemaObject
(m/schema
[:schema {:registry custom-registry}
::thing]
{:registry default-registry}))
;; Works
(def =>fn-schema
(m/schema
[:=> [:cat SchemaObject string? inst?] string?]
{::m/function-checker mg/function-checker}))
;; Fails
(def =>fn-schema
(m/schema
[:=> [:cat SchemaObject string? inst?] SchemaMap]
{::m/function-checker mg/function-checker}))
;; hangs until surfacing... Execution error (OutOfMemoryError) at clojure.test.check.generators/gen-tuple$fn (generators.cljc:89).#2023-11-3022:51dvingois ::thing a recursive schema?
https://github.com/metosin/malli/tree/master#recursive-schemas
> Without the :ref keyword, malli eagerly expands the schema until a stack overflow error is thrown:#2023-12-0120:49twashing@U051V5LLP Thanks for replying.
::thing is a nested map, where some of the leaves reference other entries in the custom-registry .
I combed through custom-registry and I don’t see anything pointing back to ::thing. Ie no circular references. But is there a tool or flag that might detect that?#2023-11-3023:05Patrick DelaneyI got excited when I saw malli.provider because I loved F# type providers, and I'm always struggling to give other readers of code concrete examples of data. It's not really a technical question I was just wondering if anybody had done anything with malli.provider that they felt particularly proud of or thought clever to share. Looking to grok whats out there#2023-11-3023:54dvingoworking with a ~8 year old datomic database which doesn't have great docs of the shape of entities. Picked a few db.unique identity attributes and queried for a large sample of them using (pull [*]) to get all first-level attributes on those entities and then pass the result to malli.provider - very useful#2023-12-0106:28Martín VarelaSimilar thing, but off a Cassandra data store. I used a bunch of query results to start off my specs.#2023-12-0115:13ikitommiRoot Reason for creating malli providers: 7y of MongoDB data accumulation, with evolutionary typos, key renames & all: with malli, got full timestamped schema evolution 💪#2023-12-0116:39Rob HaisfieldWould really love if there were a plug and play way to use Malli with a TypeScript library#2023-12-0117:24lukaszI'm looking into this - I haven't made much progress, but all the pieces are there:
• Malli can generate JSON Schemas: https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc
• There's many ways of turning JSON Schemas into Typescript types
I have a working system that uses Avro schemas that generates Typescript types, but it requires quite a lot of work to use these types in the frontend code. Based on my research and experience something like this: https://redux-toolkit.js.org/rtk-query/usage/code-generation#openapi seems to be way more maintainable .#2023-12-0119:46escherizeI haven’t used it, but there’s a library for generating typescript types from malli schemas:
https://github.com/flowyourmoney/malli-ts#2023-12-0121:33twashingIs there a fn or approach to do a Clojure-style merge on a malli.core/schema?
(def custom-registry
{::thing [:map [:a :string]]})
(def Foo
(m/schema
[:schema {:registry custom-registry}
::thing]
{:registry default-registry}))
(type (m/deref Foo)) ;; :malli.core/schema
(type Foo) ;; :malli.core/schema
;; Not working
(m/schema
[:merge
Foo
[:map [:b :string]]])
=> clojure.lang.ExceptionInfo: :malli.core/invalid-schema {:schema :merge}
{:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :merge}}#2023-12-0212:33ikitommitry malli.util/merge. there is the :merge schema, but it’s not included in the default registry, so you have to register or reference it to use it.#2023-12-1109:17SamIn all function schema examples I find, like https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schemas-1 , there's the keyword :cat . What is it? I tried to read a blogpost about seqexp which went way over my head. In the example below, it seems just fine to use [:=> :int :int] instead of [=> [:cat :int] :int] . Why is the :cat needed, what does it do?
(defn foo
{:malli/schema [:=> :int :int]}
[x]
(inc x))
(defn foo2
{:malli/schema [:=> [:cat :int] :int]}
[x]
(inc x))
(= (foo2 1) (foo 1)) ;; both work the same#2023-12-1109:51delaguardo:cat is needed when there is more than one argument expected by the function. this is the way to say "function foo expects two integers"#2023-12-1109:54SamOh, should have thought of that. Thank you!#2023-12-1210:47Alex SkyHello!
How correct is it that in regexp we cut off uuid with upper case? What do you think?
I used to be able to pass such uuid in old versions for example 0525AEAC-310A-4B2F-91EA-DD940AA1C30D
(def ^:private uuid-re
#"(?i)^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
(defn -string->uuid [x]
(if (string? x)
(if-let [x (re-matches uuid-re x)]
#?(:clj (UUID/fromString x)
:cljs (uuid x))
x)
x))#2023-12-1210:58Alex SkyStandards such as https://www.itu.int/en/ITU-T/asn1/Pages/UUID/uuids.aspx and https://www.rfc-editor.org/rfc/rfc4122 require them to be formatted using lower-case letters, but also require parsers to accept upper-case letters.#2023-12-1210:58delaguardothis regex is case insensitive. at least in clojure. but I'm not sure about cljs#2023-12-1211:00delaguardoif regex begins with (?i) it should be case insensitive#2023-12-1211:02Alex Skyyou're right. Hmm, that's freakin' weird. I'll keep looking into why it's not working for me.#2023-12-1211:15Alex SkyI have a version 0.10.4 where
(def ^:private uuid-re
#"^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$")
:melting_face:#2023-12-1211:27delaguardothat's definitely a bug)#2023-12-1218:24QuestIs it possible to schema def a :namespace "value" itself?
> (type *ns*)
clojure.lang.Namespace
Trying to write a recursive schema for a nested string map representing all loaded namespaces with this shape:
{"aleph" {"flow" {:ns aleph.flow}
"http" {:ns aleph.http}
"netty" {:ns aleph.netty}}
...}#2023-12-1218:36QuestThis schema works using a more generic match with :some
(def =>ns-tree
[:schema {:registry
{::ns-tree
[:map-of :string
[:or
[:ref ::ns-tree]
[:map [:ns :some]]]]}}
::ns-tree])#2023-12-1221:54hifumi123Assuming the value associated to :ns is always a symbol, using :fn with a predicate like (comp some? (partial find-ns))#2023-12-1301:04QuestThanks, this worked with a tweak. My value in :ns is an actual namespace "object" & find-ns can't take it
A quick call to str stops the exception -- [:fn (comp find-ns symbol str)]
I assume this was too niche or that there's pragmatic reasons that :namespace isn't a Malli builtin -- but I'm willing to PR it if anyone sees this thread in the future & wants it.#2023-12-1320:36hifumi123If you have an actual namespace object, then you can use a predicate like (partial instance? clojure.lang.Namespace)#2023-12-1320:37hifumi123e.g.
user=> (instance? clojure.lang.Namespace (find-ns 'user))
true
Minor caveat: this only tests if an object is a Namespace, it does not tell you whether the corresponding namespace has been loaded or not#2023-12-1320:41QuestI like that as a more performant and generic solution, not often I've needed instance?
Having a pred for a loaded namespace is useful but heavier & arguably dirty to include in an otherwise "pure" validation
(Sidenote: for writing a performant "is namespace loaded check", the-ns is an interesting option that takes either a ns object or a symbol naming a namespace object. I've never seen this FN in a decade of Clojure so making a note here.
https://clojuredocs.org/clojure.core/the-ns )#2023-12-1311:50SamHow do you express how schemas evolve?
For example, I validate the input to guarantee-language with this schema:
;;input to guarantee-language
[:map
[:language {optional true} :string]
[...]]
For the output schema, I would like to say it's the input schema of guarantee-lang but [:language {optional true} :string] is replaced with [:language :string] .
Since it's just vector I could do this myself in a presumably ugly way, but I wanted to know if there's a standard way to do it.#2023-12-1312:02bortexzYou can use malli.util/required-keys (mu/required-keys guarantee-language [:language])#2023-12-1312:03SamGreat, I see that malli.util has some other neat functionality for that too. Thank you!#2023-12-1313:10eval2020I wrote malli-select (linked in Malli’s README) to allow for selection of required map-attributes.
It handles maybe’s/collections etc. supports pruning (of optionals) and raises when you try to select an unknown path.#2023-12-1313:11SamThanks! This is quite the rabbit hole.#2023-12-1322:21escherize@U04V6FEES I tried out the deps-try thing (pretty cool), and use malli a lot. are you getting much use out of malli-select in practice?#2023-12-1410:57eval2020@U051GFP2V thanks. Yeah, for a pet-project I have a video-entity that starts out as just a youtube-id and gets extended in several steps (title, duration, playable?, seconds played, completed? etc.). Having one schema that shows the total picture of a video and then being able to quickly create subset-schemas for processes that do small updates is very helpful.
Also generating test-data is a breeze this way, especially with a little glue-code that injects the :gen/* properties: https://twitter.com/DontBeEval/status/1703024976947744980#2023-12-1407:25macrobartfastIs this a valid schema?
(def schema
{:cat/id :uuid
:cat [:map {:closed true}
[:xt/id :cat/id]
[:cat/name :string]
[:cat/age :int]]})
#2023-12-1410:39Stig BrautasetI don't think so? It appears you're attempting to specify a registry, but it looks like you're missing a couple nesting levels. See e.g. https://malli.io/?value=%5B1%20%5B2%20%5B3%20%5B4%20nil%5D%5D%5D%5D&schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22ConsCell%22%20%5B%3Amaybe%20%5B%3Atuple%20%3Aint%20%5B%3Aref%20%22ConsCell%22%5D%5D%5D%7D%7D%0A%20%22ConsCell%22%5D#2023-12-1410:42Stig BrautasetYou don't have to use a registry, though. You could also specify it like this:
(def CatId uuid?)
(def Cat
[:map {:closed true}
[:xt/id CatId]
[:cat/name :string]
[:cat/age :int]])
#2023-12-1607:49macrobartfastThanks! That latter option will work great.#2023-12-2115:55erwinrooijakkersafter upgrading malli and clojure and sci, I get the error :malli.core/sci-not-available when validating a schema that uses :fn, what can be wrong?
project.clj went from:
[org.clojure/clojure "1.10.1"]
[borkdude/sci "0.1.1-alpha.7"]
[metosin/malli "0.3.1"]
to
[org.clojure/clojure "1.11.1"]
[org.babashka/sci "0.8.41"]
[metosin/malli "0.13.0"]
#2023-12-2116:05borkdude@U2PGHFU5U you need to upgrade to
org.babashka/sci
#2023-12-2116:05borkdudethe organization changed like ages ago :)#2023-12-2116:06borkdudeoh sorry you did that#2023-12-2116:06erwinrooijakkershaha yes we haven’t bumped the deps in some time#2023-12-2116:08borkdude> For ClojureScript, you need to require sci.core or malli.cherry manually.#2023-12-2116:08borkdudehttps://github.com/metosin/malli?tab=readme-ov-file#serializable-functions#2023-12-2116:09erwinrooijakkersthanks!!#2023-12-2116:09erwinrooijakkersnice find#2023-12-2712:12erwinrooijakkersalso had to add [borkdude/edamame "1.3.23"] to fix
Execution error (FileNotFoundException) at edamame.impl.parser/eval40169$loading (parser.cljc:1).
Could not locate clojure/tools/reader/impl/inspect__init.class, clojure/tools/reader/impl/inspect.clj or clojure/tools/reader/impl/inspect.cljc on classpath.
but now it works 🙂#2023-12-2115:56erwinrooijakkershmm maybe I just need [borkdude/sci "0.2.7"]#2023-12-2115:59erwinrooijakkersnope, still get :malli.core/sci-not-available#2023-12-2719:28ikitommimany people have asked for better error message for malli internal errors => we can capture all calls to m/-exception in dev-mode (with !) and pretty print them. Easy to extend to cover all cases, but even with just defaults, looks better:#2023-12-2719:30ikitommiwhy doesn’t malli concatenate the ex-data into message string? pr-str is a performance killer.#2024-12-3111:11ikitommifew more screenshots (schema creation & coercion):#2024-01-0215:02Omarthat looks nice. this already in latest malli?#2024-01-0218:01ikitommijust a MR, should be out this week.#2024-12-3111:11ikitommifew more screenshots (schema creation & coercion):#2024-01-0213:26mx2000Is there a reference API for the schema keywords and options, e.g. I accidentally found
[:qualified-keyword {:namespace :foo}]
option in the README.#2024-01-0218:01ikitomminot yet, but each schema can describe it’s children and properties as malli schemas, just need to add those to all schemas. And then create documentation out of that.#2024-01-0218:01mx2000and how do I see those descriptions? Is there a function for it?#2024-01-0218:05ikitommithere are protocol functions to get those, but those all return nilat the moment -> would need to add the schemas first to all schemas. Technically to get allowed properties for :map would be:
(m/properties-schema (m/parent :map))
; => nil
… and the (empty) implementation is here: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L961#2024-01-0219:58ikitommihttps://github.com/metosin/malli/pull/981 a helper malli.util/deref-recursive:
(mu/deref-recursive
[:schema {:registry {::user-id :uuid
::address [:map
[:street :string]
[:lonlat [:tuple :double :double]]]
::user [:map
[:id ::user-id]
[:name :string]
[:address ::address]]}}
::user])
;[:map
; [:id :uuid]
; [:name :string]
; [:address [:map
; [:street :string]
; [:lonlat [:tuple :double :double]]]]]#2024-01-0308:13ikitommiEnhanced Development Mode merged in master (https://github.com/metosin/malli/pull/980). Guide here: https://github.com/metosin/malli?tab=readme-ov-file#pretty-errors#2024-01-0318:01mx2000How do I create a schema for a vector of two integers >= 0, where [val max], the second integer 'max' should always be >= val.
Example [0 0], [1 3], [5 10], [0 9]#2024-01-0318:06Stig BrautasetI don't know how to do that without the dreaded :fn schema, unless you make them distinct ranges, e.g.:
[:tuple [:int {:min 0 :max 10}] [:int {:min 10 :max 20}]]#2024-01-0318:07mx2000I do not have any ranges, just val needs to be <=max in [val max]#2024-01-0318:08mx2000I am having this val-max vector in my videogame for damage [min-max] or hitpoints [current-max].#2024-01-0318:09mx2000Maybe writing a custom 'type' ?#2024-01-0318:10Stig BrautasetMaybe...? The only other thing I can think of is:
[:and [:tuple [:int {:min 0}] [:int {:min 0}]] [:fn (fn [[a b]] (<= a b))]]#2024-01-0318:11Stig BrautasetBut I gather :fn schemas are discouraged, and best avoided if you can find a different solution.#2024-01-0318:12mx2000Why are they discouraged? I am new to malli haven't used it before.#2024-01-0318:13mx2000It would be nice to have a type :val-max because I am using the same schema at hp/mana/damage#2024-01-0318:14Stig BrautasetI haven't used Malli that much either, and can't really answer why they're discouraged. But I've seen that mentioned a few times in this channel.#2024-01-0318:14Stig BrautasetHere's another example that is an actual vector of two elements:
[:and [:vector {:min 2 :max 2} [:int {:min 0}]] [:fn (fn [[a b]] (<= a b))]]
#2024-01-0318:24Noah Bogart[:and ... [:fn ...]] is "bad" because it's opaque. you get a message that it failed because of the function and nothing else#2024-01-0318:24Noah Bogartbut if you add in an :error/fn metadata, you can customize#2024-01-0318:26Noah Bogart[:fn {:error/fn (fn [{[a b] :value} _]
(when (< b a)
(format "Expected a (%d) to be smaller than b (%d)" a b)))}
(fn [[a b]] (
#2024-01-0319:44mx2000#2024-01-0320:08Noah Bogart#2024-01-0320:09Noah Bogart#2024-01-0320:27ikitommi#2024-01-0320:32ikitommi#2024-01-0320:38ikitommi#2024-01-0407:13Ben Sless#2024-01-0407:19Ben Sless#2024-01-0407:20ikitommi#2024-01-0407:21ikitommi#2024-01-0407:21ikitommi#2024-01-0407:21ikitommi#2024-01-0407:22Ben Sless#2024-01-0407:22ikitommi#2024-01-0407:24ikitommi#2024-01-0407:58ikitommiqualified symbols are https://github.com/metosin/malli/pull/984 valid reference types:
(m/validate
[:map {:registry {:user/id :int
"user/name" :string
'user/country [:enum "UA"]}}
:user/id
"user/name"
'user/country]
{:user/id 123
"user/name" "Tiina"
'user/country "UA"})
; => true#2024-01-0410:42ikitommi@ben.sless draft for supporting Var Schema references: https://github.com/metosin/malli/pull/985#2024-01-0410:43ikitommiDidn’t check how sci works with vars.#2024-01-0410:45ikitommibeauty of Clojure:
(defn var-registry []
(reify
Registry
(-schema [_ type] (if (var? type) @type)))#2024-01-0410:54ikitommiAfter this, Vars would work as naked keys too, but I don’t think this has any use case, just a bonus feature.
(def description :string)
(m/validate
[:map {:registry {:user/id :int
"user/name" :string
'user/country [:enum "UA"]}}
:user/id
"user/name"
'user/country
#'description] ;; here1
{:user/id 123
"user/name" "Tiina"
'user/country "UA"
#'description "var"}) ;; here2#2024-01-0411:00Ben SlessLeft a comment regarding my main motivation for wanting this#2024-01-0502:36lwhortonso this is probably the wrong place to ask, but does anyone know of a malli-like lib that's implemented in native js? i've looked at options like zod , yup, joi. im not super impressed with any of them. i think ajv is probably the most data oriented, but it's tied tightly to the jsonschema spec.
am i missing any relevent options i should consider before picking a dependency?#2024-01-0515:48lukaszIf I'm not mistaken, you can convert Malli schemas to JSONSchema, that could be the way to go
I'm in the process of migrating my own solution where I have Prismatic Schema -> Avro -> Typescript types pipeline, and instead go with Malli -> JSONSchema to keep things simpler, but it's very early days#2024-01-0517:06lwhortoni'm in the sad position of not being able to use any clojure / clojurescript (yet)... but i can't imagine building anything resilient without the help of prismatic/spec or malli#2024-01-0517:08lukaszI think Zod combined with TS is what most people use, I have a somewhat narrow view into the JS world but that's I've seen in the wild#2024-01-0517:09lukaszbut yes, "types at the edges" are a necessity in Clojure, just to maintain one's sanity ;-)#2024-01-0709:42ikitommiI remember trying out yup-ast at one point - https://github.com/WASD-Team/yup-ast#2024-01-0520:14Brett RowberryI want a map schema with required key :a and for both or neither :b and :c to be present. Is that possible?#2024-01-0520:35Stig BrautasetMaybe:
[:or
[:map [:a int?] [:b int?] [:c int?]]
[:map [:a int?]]]
#2024-01-0520:35Stig Brautasetcf https://malli.io/?value=%7B%3Aa%201%7D&schema=%5B%3Aor%20%5B%3Amap%20%5B%3Aa%20int%3F%5D%20%5B%3Ab%20int%3F%5D%20%5B%3Ac%20int%3F%5D%5D%20%5B%3Amap%20%5B%3Aa%20int%3F%5D%5D%5D#2024-01-0520:36Stig BrautasetIt doesn't seem to translate properly to JSON Schema, mind you#2024-01-0520:37Brett RowberryUnfortunately no.
(malli.error/humanize
(malli.core/explain
[:or
[:map [:a int?] [:b int?] [:c int?]]
[:map [:a int?]]]
{:a 1
:c 1}))
=> nil#2024-01-0520:39Brett RowberryThis works if I’m willing to close the more minimal schema
[:or [:map [:a int?] [:b int?] [:c int?]] [:map {:closed true} [:a int?]]]#2024-01-0520:41Stig BrautasetInterestingly, that generates not the expected JSON Schema, but it does generate a correct Swagger schema 🙂
https://malli.io/?value=%7B%3Aa%201%7D&schema=%5B%3Aor%20%5B%3Amap%20%5B%3Aa%20int%3F%5D%20%5B%3Ab%20int%3F%5D%20%5B%3Ac%20int%3F%5D%5D%20%5B%3Amap%20%7B%3Aclosed%20true%7D%20%5B%3Aa%20int%3F%5D%5D%5D#2024-01-0520:42Brett RowberryWhat’s wrong with its JSON Schema?#2024-01-0522:14Stig BrautasetDoh! Nothing. I misread 😂#2024-01-0709:36ikitommi🚀 The Var-references are merged in master, also a clj-kondo https://github.com/metosin/malli/pull/987 for :fn schemas. Moved mu/deref-recursive into malli.core. Could cut out a release soon with all the stuff in. About Var references:
• Var-registry is enable by default, needs to be added manually if using mr/set-default-registry!
• Var deserialization is not enabled by default - does not work with ClojureScript and make GraalVM Native Images larger (thanks to @borkdude)
◦ can be enabled fro Clojure is one needs that
Below screenshots of using the Var Refs & error if one tries to deserialize those with malli.edn/read-string:#2024-01-0709:39ikitommiVar refs vs Var values - just like with Clojure
• refs retain the name of the reference - value can change if the var is reassigned
• values are inlined
if you use references, you can inline the values with m/deref-recursive.#2024-01-0914:50Noah BogartIs there support for checking metadata in schemas? Is that just an :fn schema?#2024-01-0919:10ikitomminothing built-in. Should not be many lines of code to do it via m/-simple-schema#2024-01-1021:38Noah Bogartcool, thanks. i'll check out -simple-schema#2024-01-0915:19Noah BogartIs it possible to use java classes as schemas? aka (mx/all-files :- [:* java.io.File] "Returns all files recursively in directory" [dir :- :str] ...)#2024-01-0919:12ikitommiNothing built-in, but would be simple to add via a new Registry Type. Please write an issue, could do this.#2024-01-1021:39Noah Bogartwill do#2024-01-0916:11Noah Bogartwhat is the difference between :sequential and :*? I'm trying to instrument a function arg that's a sequence of file objects. [dirs :- [:* [:fn #(instance? File %)]] and the anonymous function gets the whole sequence. but with [:sequential ...], the function gets each element in the sequence. i'm confused why this happens#2024-01-1006:39ikitommi:sequential maps to sequential? directly, where :* is a sequence/regex schema that can be used to define “one or more slots” in the parent sequence (`:cat` for example), e.g. it get’s inlined into the parent.
• schema: [:cat :int [:sequential :int] [:* :int]]
• accepts: [1 [2 3 ..] 4 5 ...]
when used outside of a sequence schema, by default, they work the same.#2024-01-1006:42ikitommisame for :? and :+:
[:cat [:= :enum] [:? :map] [:+ :any] accepts:
• [:enum "S" "M" "L"]
• [:enum {:default "S"} "S" "M" "L"]
… e.g. defined the schema for :enum schema 😉#2024-01-1021:38Noah Bogartoh i see, the sequence schemas get mapped to the parent. i'll try that out, thanks#2024-01-1021:36davehi there, i just upgraded from 0.8.9 to the latest for malli and now i am getting this error. is there a way i can find out what schema it is having the exception about?#2024-01-1118:46ikitommihi @UB9H5TE2X. You can ask (ex-data *e) to get more details. There should not be breaking changes in the schema syntax, but I think :or requires now at least one child:
(m/schema [:or])
; =throws=> :malli.core/child-error
(ex-data *e)
; {:type :malli.core/child-error,
; :message :malli.core/child-error,
; :data {:type :or, :properties nil, :children nil, :min 1, :max nil}}#2024-01-1118:47ikitommiwith upcoming 0.14.0, you get much better errors thanks to new dev-mode:#2024-01-1118:50ikitommiOne small addition before pushing 0.14.0 out, m/deref-recursive takes ::m/ref-key option to add the de-reffed ref-keys into schema properties:
(m/deref-recursive
[:schema {:registry {::user-id :uuid
::address [:map
[:street :string]]
::user [:map
[:id ::user-id]
[:friends [:set [:ref ::user]]]
[:address ::address]]}}
::user]
{::m/ref-key :id})
;[:map {:id :user/user}
; [:id [:uuid {:id :user/user-id}]]
; [:friends [:set [:ref :user/user]]]
; [:address [:map {:id :user/address} [:street :string]]]]#2024-01-1118:51ikitommiwith defaults:
(m/deref-recursive
[:schema {:registry {::user-id :uuid
::address [:map
[:street :string]]
::user [:map
[:id ::user-id]
[:friends [:set [:ref ::user]]]
[:address ::address]]}}
::user])
;[:map
; [:id :uuid]
; [:friends [:set [:ref :user/user]]]
; [:address [:map [:street :string]]]]#2024-01-1415:17Marcelo FernandesHello, trying out Malli and confused about this difference in behavior. I expected them to be the same. When having something such as [:and :string :not-blank] how can I avoid executing :not-blank when :string already failed?#2024-01-1416:00ikitommiHi. Currently, :and explain on all their childs:
(-> [:and [:string {:min 1}] #"kikka"]
(m/explain :kikka)
(me/humanize))
; => ["should be a string" "should match regex"]
Maybe better default would be to just to use the first one instead.#2024-01-1416:00ikitommie.g. this makes no sense:
(-> [:and
[:vector :any]
[:fn {:error/message "should be distinct"} #(= % (distinct %))]]
(m/explain :invalid)
(me/humanize))
; => ["invalid type" "should be distinct"]#2024-01-1416:01ikitommicould you write an issue out of this?#2024-01-1416:01Marcelo Fernandessure! Thanks for looking at this.#2024-01-1416:16Ben SlessWe need schemas algebra 🙃#2024-01-1513:12Tommi MartinHello everyone, I'm trying to learn Malli and I was wondering. There are logical operators for the schema syntax such as [:or [:and [:=> Are there any detailed documentation available as to how they operate?
I'm trying to write a schema for a messy data where i have a map such as
{:notification1 "stringValue"
;; several fixed keys from 1 to 24
:notification24 "stringValue"
;; Following key is user defined in an external source.
:<user-input> {;;homogenuous map of nodes
node-1-<user-input> {...}
node-2-<user-input> {...}}
}
Unfortunately I cannot modify the data structure at the source, only within my own application. Hence me trying to use Malli to transform it into a better format for my use. However I haven't been able to write a schema that successfully validates the random user input and the notification keys accurately.
Does anyone have pointers where I could look to learn how to write that schema successfully on http://malli.io for example?#2024-01-1514:12ikitommiHi, something like this?
(require '[malli.generator :as mg])
(mg/sample
[:map
[:notification :string]
[:notification24 :string]
[::m/default [:map-of {:gen/max 2} :string [:map-of {:gen/max 4} :string :map]]]])
;({:notification "", :notification24 ""}
; {:notification "", :notification24 "2"}
; {:notification "8", :notification24 "5y", "" {}}
; {:notification "Q", :notification24 "DD", "" {"S8K" {}}}
; {:notification "5O38", :notification24 ""}
; {:notification "v4W2",
; :notification24 "T",
; "9J" {"8" {}, "4rLwM" {}, "" {}, "KoQZ" {}},
; "0" {"kOro" {}, "Z" {}, "" {}}}
; {:notification "q6", :notification24 "", "2hxIz" {"e" {}}}
; {:notification "", :notification24 "", "0WVu" {}}
; {:notification "", :notification24 "egwdu"}
; {:notification "FBee", :notification24 "6UVm", "MjQ" {"M1Ig3" {}, "3O" {}, "a1t50QoD" {}}})#2024-01-1613:00Tommi MartinThis was precisely what I was looking for. I did manage to validate the incomind data using a modified version of your example. I'm a bit embarrased that I managed to miss / not understand how the m/default and map-of syntax worked from the readme.md. Thank you kindly for the answer#2024-01-1513:14EtzwaneHello
This is my first post in this channel
I am just starting out with Malli (in my first clojure project at work), and run into a simple problem I can not solve.
I want to validate a map that kan have either one key or the other. I try to do it like this, but it does not seem to work:
(def registry (merge (m/default-schemas) (mu/schemas)))
#2024-01-1513:14Etzwane(m/schema
"data"
{:registry (merge registry {"dependency" [:map
[:data map?]
[:dependencies {:optional true} [:set [:ref "dependency"]]]
[:type string?]
[:update-path [:vector string?]]]
"data" [:union
[:or
[:map [:data map?]]
[:map [:dependencies {:optional true} [:set [:ref "dependency"]]]]]
[:map [:type {:optional true} string?]]]})})
So either :data or :dependencies should be there.
I would expect something like:
[:map [:or [:foo string?] [:bar string?]]]
But that does not work either.
The whole semantics of :union, :or & :merge are a bit unclear to me. Can someone give me a pointer?#2024-01-1513:32Etzwaneso this works:
(m/schema
[:and
[:map
[:foo {:optional true} number?]
[:bar {:optional true} number?]]
[:fn
{:error/message "either use :foo or :bar"}
(fn [m] (or (some? (get m :foo)) (some? (get m :bar))))]]
{:registry registry})
but I doubt this is the idiomatic way to do this...#2024-01-1515:07ikitommicurrently, there isn’t more idiomatic way to do that. See https://github.com/metosin/malli/issues/474#2024-01-1515:08ikitommithere is :multi that can be used, depending on the requirements.#2024-01-1515:09ikitommiif someone can cook a good and simple syntax for defining the key requirements/dependencies, I'm all ears#2024-01-1515:16Etzwanethanks for the reply.
With my limited understanding so far, you can use :or either on the :map level
[:or
[:map [:foo number?]
[:map [:bar number?]]
or on the value level
[:map [:foo [:or number? string?]]
but not on the property level:
[:map
[:or
[:map [:foo number?]
[:map [:bar number?]]
[:baz number?]
That would seem pretty straightforward, but as I mentioned, I'm not sure I fully understand :or yet#2024-01-1515:41ikitommiThe entry-parser doesn't understand the :or currently. How would you differentiate it from expected map key :or?#2024-01-1515:41ikitommiit could be ::m/or not to mix up with normal keys.#2024-01-1515:43ikitommi[:map
[:or [:map [:x :int]]]
#2024-01-1515:44ikitommiis that map with :or key with map as value, or :map key with :int value#2024-01-1515:48ikitommi[:map
[::m/or [:map [:x :int]]]
would work and be good enough?#2024-01-1607:09EtzwaneI see your point. from that perspective it would perhaps have been nice to namespace all Malli keywords...
What makes me wonder, in your last example
[:map
[::m/or [:map [:x :int]]]
Why the second :map?
I would think that if you :or on this level, it would express nested maps, which is not what's happening and so a little surprising.
What I would expect is semantics like this: suppose we have an address that can either be a zip code and a house number, or an city, a street name and a house number, I would not be surprised to see it expressed like this:
[:map
[:house-number number?]
[::m/or
[:zip-code string?]
[::m/and
[:city string?]
[:street-name string?]]]]
#2024-01-1607:28ikitommiExactly. Didn’t notice that my schema had a bug in it. Correct pair would be:
[:map
[:or [:map [:tuple :int]]]
vs.
[:map
[::m/or [:map [:tuple :int]]]
, with would mean (despite being silly):
"a map with EITHER a key :map key with :tuple of one :int OR nothing, e.g {:map [1]}"
, so exactly what you described in:
[:map
[:house-number number?]
[::m/or
[:zip-code string?]
[::m/and
[:city string?]
[:street-name string?]]]]#2024-01-1607:29ikitommiimplementation of this would require a change in the map parser + internal data structure, so some amount of work.#2024-01-1607:33ikitommicould you your example (with reasoning) to the issue?#2024-01-1809:47EtzwaneSure! Glad to be of help. Tomorrow I will have time#2024-01-2207:48EtzwaneHi Tommi
I added what I hope was the gist of our discussion to the issue.#2024-01-1515:52Leo PoulsonHi all, happy monday
I have been enjoying using malli.experimental for instrumenting expected input / output shape.
I have a question about syntax. I have a lot of functions which look like this
(mx/defn my-function
[{:keys [file-type] :as env} :- [:map [:file-type :string]]]
...)
I would like to avoid having to duplicate the name of the key when writing the expected schema, and instead write something like
(mx/defn my-function
[{:keys [file-type :- :string] :as env}]
(+ 1 2 3))
however this doesn’t work, because clojure reads :- and :string as binding variables.
My questions are
• is this currently possible (i.e. labelling expected schema of a variable inside a destructuring expression)?
• if not, is this planned / is there a github issue I can track?
hope this makes sense, thanks#2024-01-1516:33ikitommiHear you, just needs to implemented. Not sure if that would be even a big thing to do, but things like precedence should be thought.#2024-01-1516:35ikitommiHere are the schematized parse tests: https://github.com/metosin/malli/blob/master/test/malli/destructure_test.cljc#L171-L287
Would you be interested in writing a minimalistic but complete expectation and attach that into an new issue of this?#2024-01-1516:36ikitommiRelated: https://clojurians.slack.com/archives/C1B1BB2Q3/p1694083268018519#2024-01-1516:41ikitommirevisited te parser code, kinda dense, but there is not much of it as it's using malli for parsing 😉 https://github.com/metosin/malli/blob/master/src/malli/destructure.cljc#2024-01-1521:56m sHi all, good evening!
I'm having trouble with the strip-extra-keys-transformer in conjunction with :and schemas:
(m/decode
[:and [:map [:y :int]] [:map [:x :int]]]
{:x 1 :y 2 :z 3}
(mt/strip-extra-keys-transformer))
strips all keys, instead of stripping only :z and keeping :x and :y.
I think this is a known issue, but the regular workaround of using :merge doesn't really work for my use case: I'm relying on :and to compose :map schemas together, which works great for documentation as I can have options on the composed schemas and use refs (and a registry) for the components (e.g. [:and {:title "composed schema"} ::component1 [:map {:title "component 2"} [:a :int]]]).
:merge's options are overridden by the last component, and I also lose refs when it's converted to json-schema.
Another issue is that I want to use strip-extra-keys-transformer to implement select (a la clojure.spec), and for the same reason, m/coerce also fails if I use it with strip-extra-keys-transformer.
I tried extending strip-extra-keys-transformer with a special case for :and, and I was able to generate the complete keyset, but when the transformer descends into the children, it eventually drops all the keys (as long as all the children have a disjunct set of keys, which they almost always do in my case).
Could someone please help me with implementing a transformer that strips extra keys correctly for [:and [:map ..] [:map ..]] schemas?#2024-01-1720:40ikitommiHmm. Interesting use case for :and! It works if as long as you don't close the maps.#2024-01-1720:41ikitommiCan't recall the code, but what happens if you set the maps explicitly open with :closed false property. Would that work?#2024-01-1720:42ikitommiHow are you using "options for the composed schema"?#2024-01-1720:43ikitommilosing refs with Json schema, could you show an example?#2024-01-1912:36m sThe issue with :closed false is that I would like to use the strip-extra-keys-transformer to implement select-schema (like metosin.schema-tools/select-schema), and that wouldn't work if I leave the maps open.
By "options on the composed schema" I mean documentation keys like :title, :description :json-schema/format etc. I'll write a better example in a bit. Update: I realize I've been using the wrong term options while I actually meant properties. Sorry for the confusion.#2024-01-1912:57m sadding as snippet as well for syntax highlight#2024-01-1913:07m sTo repeat, the two issues are:
1. When using , :merge also merges the ~options~ properties of the schemas that are being merged and doesn't retain the ~options~ properties for the root (e.g. with [:merge props a b c d], only the ~options~ properties of d will remain).
2. refs also get resolved in the process
But even if both of these behaviours would be changed in :merge, I would still want to keep using :and, because in some cases it describes the intention and the purpose of the schema more accurately than :merge.#2024-01-1717:18RyanIs there a way to get Malli to explain what may be invalid about a schema?#2024-01-1717:25ikitommiHave you tried with the latest version with the dev-mode turned on? Should give some insight on the error#2024-01-1717:26RyanI am in process of doing that as we speak, running in cljs, so setup is a little more involved but so far so good!#2024-01-1800:55Michael Gardnerwhat's the cleanest way to spec a fn that should take a map and a "type" keyword, where the expected shape of the map depends on the type?#2024-01-1800:57Michael GardnerI'm aware of both multi schemas and :fn, I'm just not sure how to combine them with :=>#2024-01-1800:58Michael Gardnerand I thought maybe there's some other, cleaner way#2024-01-1805:05ikitommiGood morning. Maybe :multi? https://github.com/metosin/malli?tab=readme-ov-file#multi-schemas#2024-01-1818:20Michael GardnerI know about multi schemas, I'm just missing the piece of how to use them with :=>#2024-01-1819:17ikitommiso, a single argument?#2024-01-1819:17ikitommi[:=> [:cat [:multi ...] ...]#2024-01-1819:29Michael Gardnerno, that's the problem— there are two separate arguments, one of which should determine the shape of the other#2024-01-1904:56ikitommiI see. Currently there is a limitation that the first argument of :=> should be :cat or :catn (https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1764-L1765). I think this could (and should) be removed.#2024-01-1904:57ikitommiIf that would be gone, e.g. one could compose schema with :and or :multi, would you be able to describe the schema you want?#2024-01-1904:58ikitommicould you write an issue out of this with example, should no be hard to fix.#2024-01-1905:02ikitommibit related, maybe these could be allowed too:
• [:=> :int :int] ~> [:=> [:cat :int] :int]
• [:=> :int :int :int :int] ~> [:=> [:cat :int :int :int] :int]
adding sugar bites in 9/10 of the cases, but maybe is this the 1/10 😅
comment welcome.#2024-01-1905:05ikitommi… or a new schema type for simple fixed arguments?
[:=>> :int :int :int :int]#2024-01-1905:06ikitommi… or just no sugar to the core lib.#2024-01-1906:37Noah BogartIf you simply remove the :cat, you have haskell/ML style type signatures. In lieu of (a, b, c) -> d syntax, i think ML syntax is cleanest and :cat syntax very verbose#2024-01-1906:38Noah BogartGiven that clojure doesn’t have multiple returns, there’s no ambiguity: everything but last is argument types, last is return type#2024-01-1906:38Noah BogartI would love this#2024-01-1906:39ikitommiwhat about varargs?#2024-01-1906:39Noah BogartHow are varargs handled right now? #2024-01-1906:42Noah BogartNaively, i would expect the vararg to be typed with :*#2024-01-1906:44ikitommi[:=> [:cat :int :boolen [:* :any]] :boolean]#2024-01-1906:45ikitommibut, I agree, extra :cat is just boilerplate in 99% of the cases.#2024-01-1906:51Noah BogartSo in that case, it would be [:=> :int :boolean [:* :any] :boolean]? That feels readable to me#2024-01-1919:24Michael Gardnercreated https://github.com/metosin/malli/issues/994. Haven't formed an opinion yet on the proposed sugar#2024-01-1905:02ikitommibit related, maybe these could be allowed too:
• [:=> :int :int] ~> [:=> [:cat :int] :int]
• [:=> :int :int :int :int] ~> [:=> [:cat :int :int :int] :int]
adding sugar bites in 9/10 of the cases, but maybe is this the 1/10 😅
comment welcome.#2024-01-1910:45MariusGood morning!
My swagger.json contains the following invalid section, where the $ref in properties does not match the definitions :
"schema":
{
"type": "object",
"properties":
{
"company":
{
"$ref": "#/definitions/lohnica.schema.validation~1company-all-optional"
}
},
"required":
[
"company"
],
"additionalProperties": false,
"definitions":
{
"lohnica.schema.validation/company-all-optional":
{
"type": "object",
"properties":
{
"name":
{
"type": "string",
"minLength": 1,
"maxLength": 90
},
(...)
Swagger UI will show
Could not resolve reference: Could not resolve pointer: /definitions/lohnica.schema.validation~1company-all-optional does not exist in document
Do you know how I can fix this?#2024-01-1913:33MariusIt works when I move the definitions into the root of the document, then the $ref is correct.
I just switched from reitit 0.6.0 to 0.7.0-alpha7 and some definitions show up in the right place (in /definitions), but now all….??#2024-01-1914:16ikitommi@U02G3DBJ2SY could you try with updating manually to latest malli? There was a fix for JSON Schema mapping + refs, not sure, but it might help here.#2024-01-1914:21Marius@U055NJ5CC I’m using [metosin/malli "0.14.0"] already#2024-01-1914:24ikitommican you paste a schema that generates that wrong ref?#2024-01-1914:26ikitommie.g. (malli.json-schema/transform schema) -> 💥#2024-01-1914:38MariusI’m using a custom registry, the part from the route spec looks like this:
:put
{:parameters {:body [:map [:company ::validation/company-all-optional]]}
:handler
My registry looks like this
(def registry
(merge
(mc/default-schemas)
{
::employee employee-schema
::employee-all-optional (optional-keys employee-schema)
::company company-schema
::company-all-optional (optional-keys company-schema)#2024-01-1914:38Mariusbtw. the ref’s work fine for employee-all-optional#2024-01-1914:40MariusAnd company-schema is quite large, similar to this
(def company-schema
[:map
[:name [:string {:min 1 :max 90}]]
[:legal-form {:optional true} [:string]]
[:bank-accounts {:optional true} [:map-of :uuid company-bank-account]]
[:contact
[:map
[:salutation {:default :none} [:enum :male :female :none]]
[:name [:string {:min 1}]]
[:phone [:string {:min 1}]]
[:email {:optional true} :string]]]
[:partial-monthly-salary-calculation-method {:optional true} [:enum :calendar-days-method :30-days-method]]
[:insolvency {:optional true} [:map [:start-date LocalDate]
[:end-date {:optional true} LocalDate]]]])
#2024-01-2319:34MariusHi @U055NJ5CC, when I call malli.json-schema/transform or malli.swagger/transform manually, schemas are generated without issues.
The problems are that
a) the swagger.json does not contain any “global”/root-level definitions - all definitions are local/relative to the methods, but the $ref points to the root definitions.
b) even in the local definitions, some are missing (without apparent reason, generating the schema manually works fine)
I remember having seen a Github issue about making all definitions local to the methods, but can’t find it any more….#2024-01-2320:31MariusFound it: https://github.com/metosin/reitit/pull/589?notification_referrer_id=NT_kwDOAIHfxrI1Nzg0MTcyMjU2Ojg1MTE0MzA#event-10256914204#2024-01-2003:19Joel(def registry
{:my/group [:vector :my/item]
:my/item [:int {:min 0 :max 480}]})
For :my/group I’d like to know that it’s a collection of :my/item’s. I tried the following
(let [k :my/group
s (some->> (m/schema [:schema {:registry registry} k])
(m/deref-all))
t (m/type s)
sub (first (mu/subschemas s))]
(if (contains? #{:vector :set} t)
{:type t :item-type (m/children (:schema sub))}))
; => {:type :vector, :item-type [:my/item]}
The :item-type is almost what I need but even though the returned [:my/item] looks like a list of one keyword, it’s actually of type :malli.core/schema
I can’t figure out, one, if I’m going about this reasonably, and two, how to get that keyword looking schema to be an actual keyword as it appears in the original schema.#2024-01-2010:38ikitommiyou can call m/form to get the data representation out:
(let [schema (m/deref-all
[:schema {:registry {:my/group [:vector :my/item]
:my/item [:int {:min 0 :max 480}]}}
:my/group])
type (m/type schema)]
(if (#{:vector :set} type)
{:type type
:item-type (-> schema (m/children) (first) (m/form))}))
; => {:type :vector, :item-type :my/item}#2024-01-2010:43ikitommiif you need to do this recursive, you can use m/walk:
(m/walk
[:schema {:registry {:my/group [:vector :my/item]
:my/item [:int {:min 0 :max 480}]}}
:my/group]
(fn [schema _ children _]
(cond-> {:type (m/type schema)}
children (assoc :children children)))
{::m/walk-schema-refs true})
;{:type :schema,
; :children [{:type :malli.core/schema,
; :children [{:type :vector
; :children [{:type :malli.core/schema
; :children [{:type :int}]}]}]}]}#2024-01-2009:26ikitommikinda nice?#2024-01-2009:27ikitommiwould like to:
• rename m/-type -> m/-name
• m/kind -> m/type (reading :type property)
… but this would be a massive breaking change.#2024-01-2009:28ikitommianyway, this gets rid of a lot of extra code in generators, tranformers, error messages and applications like json-schema#2024-01-2009:29ikitommie.g. no need to map separately all of int?, pos-int?, neg-int?, nat-int?, :int … just :int and map everything to it via the new :kind#2024-01-2009:51Ben SlessLetting users set it via property sounds like a fun source of bugs#2024-01-2010:30ikitommiDo you see some new category of errors here? Users are anyway in control of mostly everything already…#2024-01-2011:18Ben SlessYes. Adding the kind property can contradict what the code actually does. Same problem with doc strings accumulating skew but with actual behavioral effect#2024-01-2011:19Ben SlessIt's good for an internal property but letting users set it gives me nightmares#2024-01-2013:51Noah BogartWould it be possible to limit it to only :fn schemas?#2024-01-2015:55Ben Sless[:fn {:kind :string} pos?]
Enjoy finding the source of this bug in a large code base#2024-01-2019:09ikitommihow is that different to:
[:and :string [:fn pos?]]
also, many other ways to create invalid logic:
[:int {:gen/schema :string}]
[:string {:decode/string parse-long}]#2024-01-2019:13ikitommilimiting only to :fn - anything is possible, just would need reasoning for limiting just to that, e.g.
[:enum {:kind :uuid}
#uuid"f4ba0d9c-7b93-4030-8bf4-d731d5c3dc4d"
#uuid"954cb679-a36e-4b85-aaae-0844ad79a219"]
is the simplest way to describe a enum of uuids that also contains explicit type information (for decoding, encoding, clj-kondo etc)#2024-01-2019:19ikitommias the :and works already, you could say:
[:and :uuid [:enum #uuid"f4ba0d9c-7b93-4030-8bf4-d731d5c3dc4d" #uuid"954cb679-a36e-4b85-aaae-0844ad79a219"]]
… and get almost the same benefits.#2024-01-2019:21ikitommifor :type-properties, something like this is anyway good to get rid of those extras:
'double? {:error/message {:en "should be a double"}}
'boolean? {:error/message {:en "should be a boolean"}}
'string? {:error/message {:en "should be a string"}}#2024-01-2019:46Ben SlessIt isn't, right, everything is a compromise. It's also one of the reasons I'm generally averse to fn schemas. This is just what I'm paranoid about. The more subtle the detail the more interesting the bug will end up being. Maybe in the grand scheme of things it's an acceptable tradeoff#2024-01-2014:19Karol WójcikHow one can create a relation between the generated values?
For instance I want to make sure that when the :type property in map is number the generator generate only numbers in :content property of the map.#2024-01-2016:21Ben SlessDoes a multi schema not cover it?#2024-01-2020:22Karol WójcikCool. Was not aware of multischema existence. Thanks!#2024-01-2023:33Nikolas PafitisI just opened this feature request to support ClojureDart on GH, but I'd like to see some discussion here https://github.com/metosin/malli/issues/996.#2024-01-2109:15ikitommiSee https://clojurians.slack.com/archives/CLDK6MFMK/p1701196206248599?thread_ts=1701196206.248599&cid=CLDK6MFMK#2024-01-2109:15ikitommiso, multimethods is/was an issue.#2024-01-2114:30Nikolas PafitisI see. If ClojureDarts implements multimethods, and properly handle reader conditionals, it becomes relatively straight forward to enable malli in ClojureDart.
As an example in namespace malli.impl.util the following can be changed to use the :default reader conditional instead of :cljs as that functionality would be expected to be provided by any clojure implementation.
(defn -vmap
([os] (-vmap identity os))
([f os] #?(:clj (let [c (count os)]
(if-not (zero? c)
(let [oa (object-array c), iter (.iterator ^Iterable os)]
(loop [n 0] (when (.hasNext iter) (aset oa n (f (.next iter))) (recur (unchecked-inc n))))
#?(:bb (vec oa)
:clj (LazilyPersistentVector/createOwning oa))) []))
:cljs (into [] (map f) os))))#2024-01-2116:05ikitommiyes, happy to see a PR of those changes to reader conditionals if everything else works.#2024-01-2302:37tony.kayHey there. I’m relatively new to using malli, and I’ve got a regular case that I was doing with spec that I cannot figure out with the sequential schemas that malli supports. I want to validate like s/every?, but my value could be a lazy seq, list, set, vector, or even map…basically anything that you can seq. I’ve tried all of the built-in stuff I’ve found, and have resorted to making my own:
(m/-collection-schema {:type :every, :pred coll?})
I don’t see that anywhere in Malli. Should it be there? Is there something I’ve missed that does the same thing?#2024-01-2305:33ikitommiNo such thing at the moment. So, would coll? match all of those? if so, maybe a new`:collection` schema?#2024-01-2305:33ikitommi[:collection :int]
[:collection {:min 1, :max 10} :any]
[:collection {:distinct true, :min 1} :string]#2024-01-2317:06tony.kaySo I’m porting over code that uses s/every and s/coll-of, and both of those don’t care what kind of collection it is, though they have an option to help with generators (:into). every is NOT exhaustive (for performance) and coll-of is. I like them both, but having something close to either would be great. The -collection-schema I proposed above is what I installed in my registry, and it has covered the cases I’ve hit so far.#2024-01-2318:16ikitommiok. good to hear you are happy with that. I’m open to adding :collection that to malli, but it would also needs these:
• error messages
• generator
• json-schema mapping
• transforming mappings#2024-01-2318:27tony.kayah good point. I’m not deep enough into Malli yet to have used any of that except error messages 😄#2024-01-2302:38tony.kayotherwise I’m having to write things like [:or [:set :int] [:sequential :int]]#2024-01-2303:41escherizeI have a pretty flexible, recursive schema https://github.com/escherize/huff/blob/master/src/huff2/core.clj#L29-L58. Are there any tricks for getting better error messages?#2024-01-2305:34jeroenvandijkI think instead of :orn you could use : multi . This will show an error for one branch only, where as :orn will explain in the context of all branches, because malli cannot now what branch it was supposed to dispatch on.#2024-01-2305:40ikitommithe regex errors are really bad atm and should be improved (just need to figure out how, ideas welcome). here’s my test on “schemas of schemas” with :enum syntax. Simplest possible case and the error is 🙈#2024-01-2305:40ikitommithe regex errors are really bad atm and should be improved (just need to figure out how, ideas welcome). here’s my test on “schemas of schemas” with :enum syntax. Simplest possible case and the error is 🙈#2024-01-2314:15MariusHey there! Is there a way with Malli coercion to remove optional keys if their value is nil instead of e.g. an integer?#2024-01-2409:26hifumi123Nothing out-of-the-box in Malli, but using a custom decoder for your map schema, you should be able to walk map entries and ignore those with nil value.#2024-01-2409:30MariusOkay, thanks! I haven’t looked in to custom decoders, yet - might be worth a try#2024-01-2409:31hifumi123It’s a bit tricky to get right at first, but Malli does have some built-in functions for e.g. stripping extra keys. That may help guide you#2024-02-2914:14andersmurphySomething like this should work:
(def strip-nil-transformer
(let [strip-nil
{:compile
(fn [schema _]
(let [default-schema (m/default-schema schema)]
{:enter
(fn [x]
(if (and (map? x) (not default-schema))
(reduce-kv
(fn [acc k v]
(if (nil? v) (dissoc acc k) acc))
x x)
x))}))}]
(mt/transformer
{:decoders {:map strip-nil}})))#2024-03-0410:49MariusI’ll try it out, thank you!#2024-01-2318:50allentiakHi,
I'm moving from clojure.spec to malli.
I'm having it hard to instrument a function's output.
In this case, I want to make sure that the output is larger or equal than the input.
In spec, I would use
(s/fdef area-of-square
:args (s/cat :len nat-int?)
:ret nat-int?
:fn #(>= (:ret %)
(:len %)))
In Malli, I'm using defn's metadata syntax:
(defn area-of-square-with-schema
[len]
{:malli/schema [:=> [:cat nat-int?] [:and
nat-int?
(>= area length)]]}
(* len len))
I get an invalid schema exception.
How would you suggest I go about this?
Again, I want to make sure that the output ("area") is larger or equal than the input ("length").#2024-01-2318:52Noah Bogart[:and nat-int? [:fn (fn [{:keys [area length]}] (>= area length))]] should be right#2024-01-2319:24ikitommi, but - there is no way to correlate inputs & outputs currently like :fn is for spec. Should be easy to add thou.#2024-01-2319:24allentiak@UEENNMX0T Thanks for your answer, but what I'm trying to find here is a way for the output to refer not only to the input, but also to its own value.
Your suggestion assumes I have two input params (`area` and lenght); however, I only have one: length.
(`area` is the way in which I refer to the output value).
I tried the following code, with :catn (for named params).
But I get the same error:
[:=> [:catn [:len nat-int?]]
[:catn [:area
[:and
nat-int?
[:fn (fn [{:keys [area len]}]
(>= area len))]]]]]}
#2024-01-2319:24ikitommithere might be an issue for this, if not, please write one?#2024-01-2319:26allentiakThanks for your answer, @U055NJ5CC!
I found the corresponding issue:
https://github.com/metosin/malli/issues/936#2024-01-2719:45ikitommiMerged in master, see guide https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-guards#2024-01-2323:15tomcHi all, a couple Malli questions:
• would the maintainers accept a PR that adds a table of contents and perhaps slightly reorganizes the readme?
• Is there a built-in schema for anything "listy"? That is: lists, seqs, vectors, and sets. (side note, is there a term for these things that excludes maps? Basically, a collection of values, but not keyed values)
• If there's no built-in for that, is the best approach to use something like [:or [:set Schema] [:sequential Schema]] ?#2024-01-2407:02ikitommi• doesn’t github has auto-toc nowadays? see the pic.
• improving README is most welcome, big re-org - maybe, if it clearly makes things better, would need to see the changes first
• the last one, I think this was just discussed yesterday, see https://clojurians.slack.com/archives/CLDK6MFMK/p1705977455955919#2024-01-2415:41tomcThanks, this answers my questions. I guess I never noticed the TOC button on github 😄#2024-01-2412:23MariusI think I found a bug regarding Swagger Schema generation: Only definitions from schemas referred to in responses are shown in /definitions, but those from requests are missing.
Example: With the following route config
:parameters {:body [:map [:company ::validation/company1]]}
:responses {200 {:body ::validation/company2}}
only company2 will be in definitions (although the ref$to both def’s are there).
I looked through the issues on Github and could not find anything related - is this a known issue or should I open a new one?#2024-01-2913:46MariusI have submitted https://github.com/metosin/malli/issues/1002#2024-01-2415:50tomcA philosophical question about Malli, and schema checking in general: how much validation is too much? In my application, we have a datomic database wherein entities reference each other with non-component refs. In the maps I'm validating with Malli, these reference attributes have values that look like {:db/id int}. To only check that the map looks like that (has a :db/id key with an integer value) isn't quite sufficient to know the data is valid - I'd also like to check that the referenced entity is valid to be referenced from this attribute (perhaps by checking the attributes on the referenced entity). I can do this by getting access to a Datomic db instance in a :fn schema, and doing arbitrary queries. This leads to my conundrum. Should I do that? Am I setting myself up for problems later? My instinct is to include that validation code but make it conditional on whether or not there's a Datomic db in a *db* dynamic variable. Perhaps someone can convince me this is a good or bad idea.#2024-01-2415:56Stig BrautasetI'm leaning towards validating (with Malli) that the entities are of the right shape, but not validate that correctly-shaped identifiers point to valid entities themselves (i.e. exists in a DB).#2024-01-2416:09tomcThanks for the feedback. I'm going to try out the db query approach and I'll report back in a few months if it turns out I have driven myself insane#2024-01-2416:11Stig BrautasetThis has come up before in this channel, btw. You might want to try a search, in case the past recommendations were more convincing than mine 🙂#2024-01-2416:11tomcthanks, will do#2024-01-2416:51Noah BogartI would recommend against validating for valid/existing data, that will be 1) an incredible performance loss, and 2) harder to test (no generative testing, no stubs, etc)#2024-01-2416:52Noah Bogartyour integration test suite should cover that ground for you#2024-01-2415:54Karol WójcikIs it possible to deref the referenced schema fully with refs as well?#2024-01-2416:46ikitommi:refs can be recursive, so it could blow the stack.#2024-01-2417:12Karol WójcikHmm, that’s right but if the user could specify max-depth of recursion then the stack would not blow. #2024-01-2417:13Karol WójcikBut yeah, totally get the rationale! Thank you 🙏 #2024-01-2417:01ikitommi+4loc later, a quick prototype of a input-output guard for :=>#2024-01-2421:31Karol WójcikHmm, Why in function schema using {:ref "something"} in [:cat] is not allowed.
The error message: :malli.core/potentially-recursive-seqex . I don't understand why malli suggest the function schema to be potentially recursive in this case 😮#2024-01-2421:39ikitommisequence schemas inside sequences are inlined - if the :ref is recursive, it could expand to unlimited sequence:
(def Schema [:cat :int [:ref #'Schema]])
would be expanded like this:
[:cat :int [:ref #'Schema]]
... expanded to ...
[:cat :int :int [:ref #'Schema]]
... expanded to ...
[:cat :int :int :int [:ref #'Schema]]
... expanded to ...
[:cat :int :int :int :int [:ref #'Schema]]
...
#2024-01-2421:42ikitommiyou can wrap the :ref with something like :schema to mark it just to use one slot in sequence.#2024-01-2421:42ikitommifrom README:
(def Hiccup
[:schema {:registry {"hiccup" [:orn
[:node [:catn
[:name keyword?]
[:props [:? [:map-of keyword? any?]]]
[:children [:* [:schema [:ref "hiccup"]]]]]]
[:primitive [:orn
[:nil nil?]
[:boolean boolean?]
[:number number?]
[:text string?]]]]}}
"hiccup"])
(def parse-hiccup (m/parser Hiccup))
(parse-hiccup
[:div {:class [:foo :bar]}
[:p "Hello, world of data"]])
;[:node
; {:name :div
; :props {:class [:foo :bar]}
; :children [[:node
; {:name :p
; :props nil
; :children [[:primitive [:text "Hello, world of data"]]]}]]}]#2024-01-2423:00Karol WójcikWow. This really works! Thank you!#2024-01-2423:00Karol WójcikThank you as well for all of the examples. Really helpful!#2024-01-2708:17ikitommiok, better to change the guard from function into Schema object. Bit more wrapping, but supports custom error messages etc out-of-the-box:#2024-01-2708:18ikitommispec uses single map as argument with :ret and :args keys, I think it’s more clear to use positional args instead, e.g. the guard function is of type args ret -> boolean.#2024-01-2708:21ikitommiwhy? less things to remember (the keys in the map) + easier to destructure:
;; spec
(fn [{:keys [args ret]})
(< (first args) ret))
;; proposal for malli
(fn [[arg] ret]
(< arg ret)#2024-01-2709:39ikitommiok, it needs to be arity1, so:
;; spec
(fn [{:keys [args ret]})
(< (first args) ret))
;; proposal for malli
(fn [[[arg] ret]]
(< arg ret)#2024-01-2812:53ikitommifunction guards are on master now, with a small https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-guards. One other idea - this has been discussed / suggested / complained many times in the past: the :=> is not super friendly as the args need to be wrapped into :cat or :catn. Don’t want to break it, but it would be easy to add a new (optional?) function for simple function definitions. Below a 5min quick stab at it:#2024-01-2812:55ikitommiwould need a way to tag “`:->` is allowed function schema type” into malli.core, but. otherwise, that would be basically it. Nice reminder that malli is pretty easy to extend 😎#2024-01-2813:19Noah BogartThat’s very cool#2024-01-2921:12seniorI've noticed that using https://github.com/metosin/malli/blob/master/src/malli/dev.clj#L39 calls emit! which will cause https://github.com/metosin/malli#clj-kondo to be updated for functions in my code that I have attached a schema to. That results in quite a bit of churn in .clj-kondo/metosin/malli-types-clj/config.edn. The clj-kondo folks recommend https://github.com/clj-kondo/clj-kondo/blob/master/doc/config.md#importing the config, but it can be pretty distracting when it changes frequently (i.e. whenever a new function is added/changed).
Is there an easy way to prevent that churn? Or do most people just put malli-types-clj in their .gitignore?#2024-01-2921:22Noah BogartI ignore it.#2024-01-3006:10jasonjckngitignore#2024-01-3114:26RyanHi, new to Malli, and not sure I have the terminology to know what to ask, but if I have a map with a :type of either :car , :duck or :boat, that determines the schema of :details in that same map, how do I set that up?#2024-01-3114:28delaguardohttps://github.com/metosin/malli?tab=readme-ov-file#multi-schemas
multi-schema might help you there#2024-01-3114:30Ryanooh that does look like what I’m after, thanks for the pointer!#2024-02-0109:25Ryan MartinHow do I know what properties a type can have? The readme shows :min and :max for strings, :optional for maps, but I can't find other examples or the full list of valid properties. Can we also define custom properties?#2024-02-0207:35ikitommiQuick answer: no easy way to know these, there is a mechanism to describe the allowed props using malli schemas, source = where info lives now, you can add any custom properties and many support namespaces use those, .e.g generators use :gen/…. , transformers use :decode/… etc.#2024-02-0114:43Noah Bogartis it possible to get the name of a schema from a schema object? i have a function that's passed different schemas and does validation on objects with them, and i'd like to be able to log the schema name#2024-02-0207:36ikitommidepends, references are like Vars - if you pass the reference, you can ask it’s name - if you ask for the value behind the reference, you can’t.#2024-02-0207:37ikitommi… but, you can attach names to the schemas ahead of time / in the registry.#2024-02-0118:53Ryanif I have a regex schema, is there any way I can annotate it so that schemas including it can be generated (cljs, so no option to generate out of the box I think) or even better, can I supply a generator fn to the schema?#2024-02-0207:39ikitommithis should work: [#"kikka" {:gen/schema :string}], also [#"kikka" {:gen/gen …test-check-generator-goes-here…}], README has all properties that can be used.#2024-02-0118:54Ryanin this case, its for phone numbers, storing E.164 with no optional + prefix (e.g.: 1235551212, 4408001123456)#2024-02-0208:05ikitommi:gen/fmap might be usefull here#2024-02-0402:54Joel(defn x
([a] 7)
([a b] 42))
(def v [:cat :string [:+ [:alt keyword? string?]]]) ; or :vector?
(m/=> x
[:function
[:=> [:cat v] :int]
[:=> [:cat :int v] :int]])
(x ["string" :keyword])
(x 1 ["string" :keyword])
I suspect the error has something to do with not being able to determine the arity, but I can’t figure out how to fix this.#2024-02-0402:54Joel[{:type clojure.lang.ExceptionInfo
:message ":malli.core/multiple-varargs"
:data {:type :malli.core/multiple-varargs, :message :malli.core/multiple-varargs, :data {:infos [{:min 2, :arity :varargs, :input [:cat [:cat :string [:+ [:alt keyword? string?]]]], :output :int} {:min 3, :arity :varargs, :input [:cat :int [:cat :string [:+ [:alt keyword? string?]]]], :output :int}]}}
:at [malli.core$_exception invokeStatic "core.cljc" 136]}]#2024-02-0407:09ikitommitry [:vector {:min 1} [:or :string :keyword]] as second arg instead. Reson: sequence schema flattening.#2024-02-0502:48JoelThat worked, but then I can’t enforce a :string as the first item in the vector:
[:vector {:min 1} :string [:or :string :keyword]]#2024-02-0502:51JoelIt sounds like you are saying that a :cat within another :cat is problematic?#2024-02-0821:29Joel@U055NJ5CC Is this a temporary limitation?#2024-02-0408:11ikitommiWant to use Classes as Schemas? Easy to add optional ClassRegistry into malli, like there is DynamicRegistry, definetely NOT as a default thing. https://github.com/metosin/malli/issues/1007.#2024-02-0408:17ikitommimalli, the library for making schema libraries - just add your own opinions awesome🙈#2024-02-0408:44ikitommiwould it be technically easy (or hard) to add support for optional type syntax for Clojure language itself? e.g. having :- or : as a marker for “next element is a type, to be omitted from code, but available for type/schema tooling? JS land has a proposal for this (https://tc39.es/proposal-type-annotations/). See https://github.com/metosin/malli/issues/1003#issuecomment-1925637260#2024-02-0408:51ikitommisame with imaginary ide/lsp support (fade out the type-docs part so it’s more pleasant to use)#2024-02-0410:05Ben SlessHow about something similar to the new method type hint syntax?#2024-02-0413:46Karol WójcikOptional type system! Yes please 🙏 #2024-02-0413:50Noah BogartTechnically? Probably easy yes. Likely to be merged? No lol#2024-02-0510:09delaguardoisn't that kinda already there? you can add metadata to almost any symbol#2024-02-0523:20tony.kayAlso, have you checked out how we’re doing it in Guardrails? It isn’t aimed at being Clojure Typed (optional type system) but the notation of gspec is way better than inline. Metadata is great for hard-core types (like Java interop or even local loop optimizations on primitives), but if we’re talking about checking “Did I hook this function to that one correctly?” then I claim that types are not the ultimate tool for a language like Clojure(script).
https://github.com/fulcrologic/guardrails
The gspec vector has the distinct advantage of:
• Not cluttering what you’re already used to reading (not inline)
• Is actually compatible with defn, in that a freestanding vector after the arglist is just “ignorable”
I’ve also made decent inroads into making a type checker from there that can do things like show you the inferred “type-like” value of a symbol in things like a let.
I don’t personally thing the “types” as a core Type System are very useful in Clojure. What is useful are data specifications ala spec/malli…and those are most definitely not formal types, but more like data contracts.
https://github.com/fulcrologic/guardrails?tab=readme-ov-file#gspec-advantages#2024-02-0604:13Karol WójcikGuardrails with malli. Day starts good! Thank you @U0CKQ19AQ for this beautiful piece of software!#2024-02-0604:23tony.kayyeah, we haven’t announced the release yet…just finishing up some output enhancements…but I’ve been using it for a few days.#2024-02-0606:28ikitommiLooking forward to announcement of 1.2.0 @U0CKQ19AQ! People have been asking for ghostwheel syntax for malli (there is https://github.com/teknql/aave, but not maintained) but as it’s sugar on top of the core, I think it should be a separate library. For same reason, the Plumatic/inline syntax is under malli.experimental, not sure where it should go. Plumatic doesn’t support inline types with key destructuring, asked many times, original reason for this thread, see https://github.com/metosin/malli/issues/1003#issuecomment-1925637260.
Quick comments about Guardrails & malli:
1. you get 10-100x perf if you can cache the validator & explainer here: https://github.com/fulcrologic/guardrails/blob/main/src/main/com/fulcrologic/guardrails/malli/core.cljc#L92-L93. e.g. (m/validator schema) returns an optimized and pure function. m/validate re-creates (and forgets) that for each time it’s called (unless the first argument is a Schema object, in case it caches validator and explainer internally, but still slower than m/validator & friends
2. mutable registry. I know many people (me included) define their own registry at application/system-level, using malli.registry/set-default-registry!, with default schemas + a mutable registry backed by a project level atom. Maybe add a way to use this in guardrails? e.g. com.fulcrologic.guardrails.malli.registry/set-registry-atom! or similar
3. malli dev-mode, malli function schemas and emitting clj-kondo definitions, would be great if guardrails followed those, e.g. ! would enable this
4. happy to help with these#2024-02-0606:35ikitommie.g. this is a the way for spec-like registry today with malli:
(require '[malli.core :as m])
(require '[malli.registry :as mr])
(def registry (atom {}))
(defn register! [type schema]
(swap! registry assoc type schema))
(mr/set-default-registry!
;; linear search
(mr/composite-registry
;; core schemas
(m/default-schemas)
;; to support Var references
(mr/var-registry)
;; mutable (spec-like) registry
(mr/mutable-registry registry)))
, registering schemas here doesn’t add them to scope of guardrails, so you need to register them twice.#2024-02-0606:43ikitommi… how to fix that:
1. use m/default-registry var in guardrails, which points to whatever user has defined as default-registry, e.g. (mr/composite-registry m/default-regisrtry ..guardrails-atom-which-starts-empy..) -> anything you have registered elsewhere is visibile to guardrails, but not the other way around
2. add com.fulcrologic.guardrails.malli.registry/set-registry-atom! which allows user to override the atom in guardrails so that both ways (project-spesific register and guardrails def> register to same place
3. add a “shared mutable atom” + register-function into malli that everyone should us, instead of defining their own mutable store per application, guardrails could use that too#2024-02-0606:50ikitommiabout adding metadata to symbol, works for symbols, keywords, strings and maps only, e.g. this works:
(let [{:keys [^:int x ^:int y]} {:x 1, :y 2}]
[x y])
but for a inlined vector schema, you need to wrap it into:
(let [{:keys [^:int x ^{:type [:tuple :int]} y]} {:x 1, :y 2}]
[x y])
which is really bad imo, also, misusing existing features.#2024-02-0606:53ikitommimx/defn and ghostweel defn> are both good, haven’t made my mind which I prefer more 🙂#2024-02-0606:57ikitommiIF there would be “optional types as docs” extension to Clojure syntax, it would open up problem: one could use all of plumatic, spec or malli schemas there, e.g.
;; plumatic
(defn kikka : Long [x : Long, y : Long] (+ x y))
;; spec
(defn kikka : int? [x : int?, y : int?] (+ x y))
;; malli
(defn kikka : :int [x : :int, y : :int] (+ x y))
, which would be really confusing for everyone.#2024-02-0607:01tony.kayTo me the two serve different purposes. Type annotations as metadata are info the compiler can make use of to write more optimal code (e.g. no use introspection or use primitives). The use of formal types in clojure IMO is otherwise useless, since most things are either primitive, maps, sets, vectors, or lists. That gives you a little insight, but not a lot. Putting schema on metadata is just too noisy. I want to see arity and names of args clearly. Now, it would be nice if the docstring showed the arglist AND under that the types of the args and the return type (the gspec). Then you’d have both.#2024-02-0607:03tony.kayRE: the registry. Remember that library authors are going to use this registry, and they may load before user code. If the application code then overrides the registry, they have to comb through libs and figure the whole mess out. No, the GR registry is for GR, and you need to merge your schemas that you want to use in gspec into what GR has defined otherwise lib composition falls apart, doesn’t it? Thus, you’d instead merge your global default INTO GR’s registry. That way your global data reg doesn’t end up with a ton of library-related schemas, and you don’t accidentally muck up those library schemas. Of course, lib authors better ns their schemas well 🙂#2024-02-0607:04tony.kayI really thought hard about the problem, and having the user be able to change out the registry seems a bad idea. Just merge the things in you need and then also benefit from what your library authors have also provided for use with their GR functions.#2024-02-0607:06tony.kayAs far as optimization: I did not know about the validator vs. validate. Yes, I’ll plan on making that optimization in a delta release soon.#2024-02-0607:09tony.kayWe did tinker with malli dev mode and emitting those, but decided not to do that for now. If someone wants to maintain such an add, we can ask @U9S6X97KQ if he still has what he wrote for that.#2024-02-0607:37ikitommi> To me the two serve different purposes. Type annotations as metadata are info the compiler can make use of to write more optimal code (e.g. no use introspection or use primitives). The use of formal types in clojure IMO is otherwise useless, since most things are either primitive, maps, sets, vectors, or lists. That gives you a little insight, but not a lot. Putting schema on metadata is just too noisy.
fully agree. I was not proposing to use meta-data just because of this, it was just proposed earlier on this thread.
Having extra marker : and just ignore the next token (like TC39), that I would not object.#2024-02-0609:43delaguardo> about adding metadata to symbol, works for symbols, keywords, strings and maps only
I don't get it. I did not suggest using existing :type metadata but add something malli specific:
(defn plus
^{:malli/schema :int}
[^{:malli/schema :int} a
^{:malli/schema :int} b]
(+ a b))
it is already there, you don't need language support for that. and it is possible to simplify using a macro#2024-02-0614:26gnlCouple of unstructured notes on the general discussion:#2024-02-0614:26gnl• Regarding a possible syntax for spec/schema/type annotations – what I would like to see is a single, universal, widely adopted syntax that can serve as a basis for both function call validation at runtime, as well as any kind of possible static type analysis. To me, both metadata and interspersed : <type> annotations have various issues with regard to succinctness, noisyness, not having a good solution for such-that predicates, getting particularly messy with multi-arity functions, etc.
#2024-02-0614:26gnlFrom my – obviously rather biased – perspective only gspec https://github.com/fulcrologic/guardrails?tab=readme-ov-file#gspec-advantages I care about (most of them objective, some less so), and, as already mentioned, has the added bonus of being perfectly compatible with regular defn syntax – no need to worry about IDE/editor support, can easily flip >defn to defn and it just works, etc.#2024-02-0614:26gnl• Currently Guardrails' >defn requires a gspec vector, but if gspec were to be used in core's defn or any other macro that has to support both specced and unspecced functions, it would require determining whether the first body form is just a regular one or a gspec vector, in which case it should be appropriately validated for possible syntax errors. The way I imagine this would work is that if the first body form in a multi-form function body is a vector, that's a gspec and treated accordingly, otherwise it's evaluated regularly. This should be reliable, as there's otherwise no legitimate reason to have a vector in there, because it's not returned and doesn't do anything, unless you use nested side-effecting calls inside of it, which – don't.#2024-02-0614:27gnl• Guardrails already emits a native Malli function schema in the metadata of the generated defn (`{:malli/schema ...}`), so if anyone wants to use Malli instrumentation for the generation of clj-kondo type annotations (or really any kind of native or third-party tooling that works with Malli function schemas), they can easily do so. As @U0CKQ19AQ already mentioned we ultimately decided against emitting a clj-kondo config automatically, but the basic implementation of it is on https://github.com/gnl/guardrails/blob/gnl/type-experiments/src/main/com/fulcrologic/guardrails/malli/core.cljc#L85. It also still has the code for the generation of JSDoc Closure type annotations (left over from Ghostwheel), which we removed in the new Guardrails release.#2024-02-0614:27gnl• Thanks @U055NJ5CC for the validator/performance hint – neither of us had much experience with Malli before we started implementing it in Guardrails, so this is very helpful.
#2024-02-0614:27Noah Bogartis there a reason guardrails doesn't use the attr-map?#2024-02-0614:28gnlDo you mean for defining function specs or in general? We use it for setting function-specific configuration options.#2024-02-0614:31gnlThere are multiple reasons we don't use it for function specs, mostly having to do with not being able to handle predicates + symbol references/refactoring as concisely and smoothly as we can with the gspec vector, and it being a huge mess when you're dealing with complex multi-arity functions. See the https://github.com/fulcrologic/guardrails?tab=readme-ov-file#gspec-advantages for more details, there are multiple things on there you can't do with the attr-map.#2024-02-0614:36gnlProbably the easiest way to illustrate this is to have a multi-arity function, specced-out with argument and return-value predicates, as well as differing return specs per arity, and macroexpand the >defn on that to see the monstrosity that is generated as a native clojure.spec fdef (we'd have to do something similar if we had to put the whole function spec in a single place like the attr-map).
You could use some of the multi-arity https://github.com/gnl/ghostwheel.specs/blob/master/src/ghostwheel/specs/clojure/core.cljc as an example.#2024-02-0614:44Noah Bogartthanks for the details, that's a compelling case#2024-02-0615:00Noah Bogartoh i said "attr-map", but i was actually thinking of the prepost-map, which goes inline after the argument vector of each fn body. i'm guessing the reasoning is the same, tho#2024-02-0615:06gnlI would also add that in multi-arity functions I like the spec/schema/type being in the same place as the arg vector, rather than having to track down the correct arity in a separate place (whether it's attr-map, s/fdef, m/=> or whatever).#2024-02-0901:08ambrosebsHere's how Typed Clojure achieves this. It combines the insights already mentioned that Clojure already supports this via metadata, and that you need namespaced keywords to distinguish different backends https://github.com/typedclojure/typedclojure/blob/main/example-projects/zero-deps/src/typed_example/zero_deps.cljc#2024-02-0511:05hanDerPederany reason there is not an encoding variant of coercer?#2024-02-0511:12ikitommino-one has asked for it#2024-02-0511:12hanDerPederConsider it asked for 🙂#2024-02-0511:14ikitommiplease write an issue#2024-02-0511:34AbhinavHi, in clojurescript how can I convert a schema object to a readable schema?
a function that takes
#object[malli.core.t_malli$core146211]
and returns something like this
=> [:map [:a :int]]
#2024-02-0511:43ikitommitry m/form on it#2024-02-0511:44ikitommialso, if there is a way to add default printing method that does that, would like to hear it#2024-02-0511:44ikitommiworks on clojure (via print-method)#2024-02-0511:53Abhinav> also, if there is a way to add default printing method that does that, would like to hear it
I'll ask around. thanks very much !#2024-02-0615:46steven kentHello, I'm trying to use a :fn schema that uses a function that is locally defined in the same file.
(*def* file-test2 [:fn '(*fn* [s] (test-fn s))])
When I try to validate using this schema an sci exception is thrown Could not resolve symbol: test-fn
I've tested test-fn in the REPL and it works as I expect it to. How to get Malli to recognize the symbol? Thank you#2024-02-0615:46borkdudeJust don't use the quote#2024-02-0615:47borkdudeunless you want to serialize the function#2024-02-0615:51steven kentThank you#2024-02-0618:39zeitsteinI have a very basic question. Say I have a set user-roles used in my code. In malli this would be e.g. (def UserRoles (into [:enum] user-roles)). If I want to avoid duplication, do I handle this as described or do I go the other way around: use the malli schema as the source of truth then either use m/validate (don't think it can replace a set for every use case) or (def user-roles (set (rest UserRoles))) ? It's just data, so both work, but wondering how you approach this.#2024-02-0619:42Franco GasperinoI've applied them into the schema.
(def user-roles [1 2 3])
(def schema-1
[:role (into [:enum] user-roles)])
also allows variations...
(def schema-2
[:role (into [:enum] (conj user-roles 4))])
#2024-02-0620:47tony.kayMalli support in Guardrails has been released. See #C06MAR553#2024-02-0721:31bzmarianoHello, today I was using Malli and I tried to validate a UUID using (malli.core/validate :uuid "caa71a26-5fe1-11ec-bf63-0242ac130002"). However, it returned false. The string is from an example on the Malli GitHub page (under the title "decoding schema in inferring") , and I've also tried a couple of other valid UUID strings and using uuid? instead of the keyword. Does anybody have an idea what the problem could be?#2024-02-0721:37Stig Brautasetit's not a UUID, it's a string representation of a UUID.#2024-02-0721:38Stig BrautasetTry (malli.core/validate :uuid #uuid "caa71a26-5fe1-11ec-bf63-0242ac130002")#2024-02-0721:38Stig Brautasetit fails for the same reason that "1" is not an integer, it's a string representation of an integer.#2024-02-0722:31bzmarianoRight, the same happened to me with java dates types and I just forgot about that 😅 #uuid works great. Thanks for the help!#2024-02-0722:40bzmarianoif I have [:map [:id :uuid] [:age :int]], Is it possible to insert the #uuid in the map schema ?#2024-02-0722:45Stig BrautasetI would recommend playing around using this tool: https://malli.io/?value=%7B%3Aid%20%23uuid%20%2246286bb1-3734-402f-8d8f-36f816d63367%22%2C%20%3Aage%200%7D&schema=%5B%3Amap%20%5B%3Aid%20%20%3Auuid%5D%20%5B%3Aage%20%3Aint%5D%5D#2024-02-0722:45Stig BrautasetYou can see there that {:id #uuid "46286bb1-3734-402f-8d8f-36f816d63367", :age 0} conforms to the map schema you gave#2024-02-0722:47bzmarianoCool, didn't know about malli playground. Thanks.#2024-02-0920:34Noah Bogartthere's :tuple which expects a vector. What if I expect a list of a specific size? what's the best way to check that?#2024-02-0921:36Stig BrautasetMaybe :sequential? cf https://malli.io/?value=undefined&schema=%5B%3Asequential%20%7B%3Amin%202%20%3Amax%202%7D%20int%3F%5D#2024-02-0921:36Noah Bogartah yeah, i forgot about the properties, thanks#2024-02-1214:44Felix DornerStraight from the quickstart:
(def my-schema
[:and
[:map
[:x int?]
[:y int?]]
[:fn (fn [{:keys [x y]}] (> x y))]])
(m/validate my-schema {:x 1, :y 0})
; => true
(m/validate my-schema {:x 1, :y 2})
; => false
The :fn schema assumes both x and y are present in the map, but what if x is missing? Do I need to check for that again in my function?#2024-02-1215:07Noah Bogartthe :and short-circuits, so if x or y aren't present or they're present but not ints, the :fn schema won't be checked#2024-02-1215:07Noah Bogartthat's why the :fn schema can just call (> x y) without additional checks#2024-02-1215:10Felix DornerThat makes sense. I was then maybe just confused by the result of explain when running without y:
{:schema
[:and [:map [:x int?] [:y int?]] [:fn #function[cookidoom-clj.schema/fn--16490]]],
:value {:x 1},
:errors
({:path [0 :y],
:in [:y],
:schema [:map [:x int?] [:y int?]],
:value nil,
:type :malli.core/missing-key}
{:path [1],
:in [],
:schema [:fn #function[cookidoom-clj.schema/fn--16490]],
:value {:x 1},
:type nil})}
#2024-02-1215:11Noah Bogarthmm maybe explain checks everything? i'm not sure about that#2024-02-1215:12Noah Bogartthe first error in the list makes sense: :path [0 :y] is :type :malli.core/missing-key#2024-02-1215:18Felix DornerHmm, but actually I would like to be able to define multiple rules on the same path. For example: name -> must be between 1 and 8 characters. and then another one name -> cannot be in the list of swearwords. Maybe this is too domain/semantic level for malli? Maybe there's better solutions for business rules?#2024-02-1215:20Felix DornerI've always been confused between schema and domain and when does one end and the other start.#2024-02-1215:35Noah Bogarti think those are fine, you just have to write them out#2024-02-1215:35Noah Bogartyou can attach specific error messages to :fn schemas if you'd like too#2024-02-1215:37Noah Bogartwe have this schema in our top-level schemas
::var [:fn {:error/fn (fn [{:keys [value]} _] (str "Expected var, given " (type value)))} var?]
#2024-02-1215:39Noah Bogartso you could say something like
[:fn
{:error/fn (fn [{:keys [value]} _] (str "Must be between 1 and 8 characters, given name of length " (count value)))}
(fn [value] (#2024-02-1215:40Noah Bogart#2024-02-1218:38Ben Sless#2024-02-1312:26Felix Dorner#2024-02-1323:28Felix DornerHow can I change [:float {:min 0 :max 1}] so that it's valid for 0 and 1 and anything between? With this it's not valid for 0 or 1. And :number instead of :float gives "invalid schema"#2024-02-1409:00ikitommioh, there is no :number at the moment, should be. 1 is not a double? in clojure (and by so, not a :double in malli)#2024-02-1409:01ikitommithis work, btw:
(map (m/validator [:and [:>= 0] [:<= 1]]) [0 0.1 1])
; => (true, true, true)
#2024-02-1409:01ikitommibut, :number would be good to have. Please check if there is an issue for it, comment on it or create new issue if doesn’t exist.#2024-02-1409:06Felix DornerOk yes makes all sense. I will check#2024-02-1410:41Felix DornerAs a beginner, I also got confused about core predicates vs built in schemas, like :stringvs string? , as there's a pitfall:
(m/validate [:string {:min 1 :max 3}] "a street") ;; false
(m/validate [string? {:min 1 :max 3}] "a street") ;; true!
Not sure this is how it's supposed to be?#2024-02-1412:14ikitommifunctions like string? are mapped to simple predicate schemas, which don’t understand any extensions like :min & :max. Thinking how to move forward:
1. map them to real schemas if they exist, .e.g string? = :string
2. remove default support of predicate schemas (could be added separately, but “not recommeded”)
3. finish the schema-of-schemas so one would get warning of unmapped keys
4. document this
5. ☕ #2024-02-1413:13Felix Dorner👍 #2024-02-1415:03DrLjótssonI would love it if you kept string? and such as valid schemas, both because I like them but also because they are everywhere in my codebase. Would be awesome if the were mapped to existing real schemas though. #2024-02-1918:06vemvKind of a silly q, but perhaps I'll get a pleasant surprise :)
I don't like typing e.g. :map much because if I don't remember some nuance about its 'syntax', then I have to visit the malli readme or some other static documentation source.
For me traditionally that doesn't feel very productive, or even Clojurey (in the Lisp sense of introspectable programs). It also doesn't help that :map is an unqualified keyword.
Then again, data is awesome, and by now I'd pick Malli before Spec 9 in 10 times 😄
Wondering if there are simple wrappers like defn map (returning a :map) which would bridge the world of data with the world of docstrings, source code, and the like. Official or otherwise.#2024-02-1918:20valtteriMalli itself could provide that information quite easily if we had meta-malli for malli-schemas. 😉 It’s been coined a few times but not done yet#2024-02-1918:21valtteriYou could perhaps even generate defns and docstrings from that#2024-02-1918:24vemvWhat would it look like?#2024-02-1918:25valtteriIt’d just be a malli-schema that describes what schemas can contain#2024-02-1918:25valtteri“schema of schema”#2024-02-1918:26vemvSure :) thanks
didn't know those didn't exist#2024-02-1918:28valtteriNope, there’s just a parser afaik#2024-02-1919:46ikitommiMali lite has helper functions - https://github.com/metosin/malli/blob/master/src/malli/experimental/lite.cljc#2024-02-1919:47ikitommiIn action: https://github.com/metosin/malli?tab=readme-ov-file#lite#2024-02-1919:50vemvNice! I'll play with it - perhaps at some point I'll want to offer a PR with docstrings or arglists.#2024-02-1918:52QuestI accidentally walked into a nasty behavior from a bad Malli spec last Friday,
on interpretation, this caused infinite recursion until my heap was exhausted
(def =>feature (m/schema [:map
[:desc :string]
[:enable? {:optional true} [:=> [:cat] some?]]]))
(def =>features (m/schema [:map-of :keyword [=>feature]]))
The bad spec is in the 2nd def at the end of =>features, note the vector wrapped [=>feature]. I'm not sure why this triggers an infinite or if it's a reasonable thing to guard against.
This was a surprise though, first time I've seen a Malli spec lock up my JVM at interpret-time.
The library is generally a joy to work with but this broke my app init & took a few minutes to fix, so FYI#2024-02-2203:58Joel[:schema {:registry registry} attribute]
Does attribute have to be a qualified-keyword?#2024-02-2207:06ikitommi(m/validate
[:schema {:registry {"kikka" :int
:kukka :int
::kakka :int
'kokka :int
`kekka :int}}
[:tuple "kikka" :kukka ::kakka 'kokka `kekka]]
[1 2 3 4 5])
; => true#2024-02-2207:06ikitommiso, no.#2024-02-2207:08ikitommiif you use references, it has type constraints:
((requiring-resolve 'malli.dev/start!))
(m/schema [:ref :kukka])
; =>
-- Schema Error ------------------------------------------------- Thread:1583 --
Invalid Reference
[:ref :kukka]
Reason
Reference should be one of the following
- a qualified keyword, [:ref :user/id]
- a qualified symbol, [:ref 'user/id]
- a string, [:ref "user/id"]
- a Var, [:ref #'user/id]
More information
--------------------------------------------------------------------------------
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
:malli.core/invalid-ref
#2024-02-2206:56BBTiger Michael@ikitommi Hi,I feel wonderful to read https://www.metosin.fi/blog/transforming-data-with-malli-and-meander and I have a question about what tool was used to get this picture.#2024-02-2207:10ikitommithanks! can’t recall, but first thing that the google found, was not a good experience, was throwaway code. Today, I would use https://reactflow.dev/ for that.#2024-02-2213:33SamIs there a way to make this work? Based on https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schemas-1
(=> foo [:=> [:cat :int] :int [:fn (fn [[arg] ret] (= 10 ret))]])
(defn foo [x]
10)
;; errors with Schema has 3 children, expected {:max 2, :min 2}#2024-02-2213:48juhoteperiSeems like tests don't have any test cases for that, so I wonder if the doc just has a issue mentioning a non existing feature#2024-02-2213:48juhoteperiOh that line was added just last month to the docs#2024-02-2213:49juhoteperihttps://github.com/metosin/malli/commit/e34e1fbf6a6bc5b4f136e12b2fe2bd693ea974ea Unreleased#2024-02-2213:49juhoteperiYeah that feature isn't on 0.14#2024-02-2214:10SamCool! Something to look forward to 🙂#2024-02-2313:22Tommi MartinSpeaking of the transforming data with malli and meander post. I found a situation where integer numbers that were strongly on the negative broke the the meander query adding min 0 to all ints restored it. in practice this showed up as real data working but malli generator data breaking when it was posted into the transformer. Has anyone else had this kind of experience or should I start to get worried? 😄#2024-02-2313:32ikitommisound really strange. do you have a repro of this?#2024-02-2313:49Tommi MartinActually no. I reverted the changes to my input and output schemas to get you the repro, but it wont happen again. I have to dig into this more now.
If you want I can give you the transformer and the schemas though#2024-02-2313:55Tommi MartinUpdate: jackout and jack-in seems to cause it to happen with unmodified schema... here is the schemas and transformer:
Input schema:
(def schema-rest-status-simple
[:map
[:routers {:optional true} [:vector [:map [:name :string] [:connections [:map [:created :int] [:active :int]]]]]]
[:composite :boolean]
[:name :string]
[:type :string]
[:policy :string]
[:datasources
[:vector
[:map
[:role :string]
[:groupId :int]
[:standby :boolean]
[:witness :boolean]
[:alertMessage :string]
[:name :string]
[:dataserver [:map [:state :string]]]
[:state :string]
[:replicator
[:map
[:role :string]
[:appliedLatency :double]
[:pipelineSource :string]
[:maxStoredSeqno :int]
[:state :string]
[:relativeLatency :double]
[:seqno :int]
[:minStoredSeqno :int]
[:appliedLastEventId :string]
[:version :string]]]
[:alertStatus :string]
[:manager [:map [:state :string]]]
[:lastError :string]
[:lastShunResult :string]
[:connections [:map [:created :int] [:active :int]]]
[:archive :boolean]]]]
[:coordinators [:vector :string]]
[:multimaster :boolean]])
output
(def schema-rest-outbound
[:schema
{:registry {"Root" [:map
[::m/default [:map-of {:gen/min 1 :gen/max 100} :keyword "Cluster"]]]
"Cluster" [:map
[:id [:string {:default "test"}]]
[:name :string]
[:isComposite :boolean]
[:policy :string]
[:topology :string]
[:dataservices [:map-of {:gen/min 1 :gen/max 4} :keyword "Dataservices"]]]
"Dataservices" [:map
[:id [:string {:default "test"}]]
[:name :string]
[:policy :string]
[:type :string]
[:coordinators [:vector :string]]
[:datasources [:map-of {:gen/min 1 :gen/max 6} :keyword "Datasources"]]]
"Datasources" [:map
[:name :string]
[:role :string]
[:state :string]
[:standby :boolean]
[:archive :boolean]
[:witness :boolean]
[:alert "SourceAlert"]
[:replicator "Replicator"]
[:manager "Manager"]
[:connections "Connections"]
[:dataserver "Dataserver"]]
"SourceAlert" [:map
[:status :string]
[:message :string]
[:lastError :string]
[:lastShunReason :string]]
"Replicator" [:map
[:appliedLatency :double]
[:relativeLatency :double]
[:seqno :int]
[:minStoredSeqno :int]
[:maxStoredSeqno :int]
[:pipelineSource :string]
[:appliedLastEventId :string]
[:version :string]
[:role :string]]
"Manager" [:map
[:state :string]]
"Connections" [:map
[:created :int]
[:active :int]]
"Dataserver" [:map
[:state :string]]}}
"Root"])
Pattern and expression
(def rest-standalone-transformation
{:registry {:rest-status schema-rest-status-simple
:dashboard schema-rest-outbound}
:mappings {:source :rest-status
:target :dashboard
:pattern '{:name ?name
:composite ?composite
:type ?type
:policy ?policy
:dashboard-id ?dashboard-id
:coordinators ?coordinators
:datasources [{:name (me/and !name !index)
:manager {:state !manager-state}
:dataserver {:state !data-state}
:role !role
:state !state
:standby !standby
:archive !archive
:witness !witness
:alertStatus !alert-s
:alertMessage !alert-m
:lastShunResult !shun
:lastError !error
:connections !connections
:replicator {:state !repl-state
:version !version
:appliedLatency !a-latency
:relativeLatency !r-latency
:seqno !seqno
:minStoredSeqno !minseqno
:maxStoredSeqno !maxseqno
:appliedLastEventId !eventId
:pipelineSource !source
:role !repl-role}} ...]}
:expression '{(me/keyword ?dashboard-id)
{:name ?name
:id ?dashboard-id
:isComposite ?composite
:topology ?type
:policy ?policy
:dataservices
{(me/keyword ?dashboard-id)
{:name ?name
:id ?dashboard-id
:type ?type
:policy ?policy
:coordinators ?coordinators
:datasources
{& ([(me/keyword !index)
{:name !name
:role !role
:state !state
:standby !standby
:archive !archive
:witness !witness
:manager {:state !manager-state}
:dataserver {:state !data-state}
:connections !connections
:replicator {:version !version
:state !repl-state
:appliedLatency !a-latency
:relativeLatency !r-latency
:seqno !seqno
:minStoredSeqno !minseqno
:maxStoredSeqno !maxseqno
:pipelineSource !source
:appliedLastEventId !eventId
:role !repl-role}
:alert {:message !alert-m
:status !alert-s
:lastError !error
:lastShunReason !shun}}] ...)}}}}}}})
Matcher function: note it has been modified:
(defn matcher
[{:keys [pattern expression]}]
(eval `(fn [data#]
(let [~'data data#]
~(mre/compile-rewrite-args ; modified: original did not use compile-rewrite-args
(list 'data pattern expression)
nil)))))
Transformer, notice the output of a map over a vector.
(defn transformer
[{:keys [registry mappings]} source-transformer target-transformer]
(let [{:keys [source target]} mappings
xf (comp (map (m/coercer (get registry source) source-transformer))
(map (matcher mappings))
(map (m/coercer (get registry target) target-transformer)))]
(fn [data] (into {} xf data))))#2024-02-2313:55Tommi MartinSorry for the wall of text, it's my first proper meander thing and it is quite... verbose still to use it, i was running this:
(def status->dashboard-status
(transformer
rest-standalone-transformation
(mt/transformer
(mt/string-transformer))
(mt/transformer
(mt/string-transformer)
(mt/default-value-transformer))))
(try
(status->dashboard-status [(assoc (mg/generate schema-rest-status-simple) :dashboard-id "dummy")])
(catch Exception ex
(-> ex ex-data :data :explain merr/humanize prn)
(prn (ex-data ex))))#2024-02-2314:08Tommi Martinmaybe it's better to not go into that code @U055NJ5CC. I'm feeling this is a tool issue and not actually related to malli:
If I jackout and jack in with calva in cursor., that schema does not work with Malli generator and the provided code.
If i fix the code style issue in schema-rest-status-simple ie routers map being on one line.
then jackout and back in again and evaluate everything, it works. So it's not actually a number range issue. It's a linebreak issue which makes me believe that there is no way it's nothing else than tool related#2024-02-2318:02Tommi MartinI figured that one out it was because of my own stupidity. There was a dependency error that caused the nested usage of meander to not work in the patterns and expressions as it was expected. hence the problems.
While fixing that problem I think I finally clicked with Malli after playing with the generator and schema interaction as I debugged. I can basically develop my entire app with the generator. This thing is incredibly powerful.
Thank you so much for making it, and making it open source.#2024-02-2313:24Tommi MartinIntrestingly, using meander directly worked but the matcher + transformer syntax broke that usecase with the negative numbers#2024-02-2317:03Henrik LarssonI am trying to introduce Clojure at work since I think I have found a perfect candidate for refactor.
We have a legacy system that save lots of expressions in a database in the form
"~a{b=ii}|c{d=u}&~(e{f='[a-z]'}|a{b=3})"
this basically describe the constraints over values in a map. Where a{b} is just the path in the map. Now what I have done
is writing a parser where I parse each of these expression and compile them into Malli like this (this is not what the above expression produce just an example of data):
'([:and [:not [:map ["a" [:map ["b" [:re #"dd[A|B|C|F|D|E]"]]]]]] [:or [:map ["c" [:map ["d" [:= "3"]]]]] [:map ["c" [:map ["d" [:= "2"]]]]] [:map ["c" [:map ["d" [:= "9"]]]]] [:map ["c" [:map ["d" [:= "99"]]]]]]]
[:not [:map ["d" [:map ["e" [:re #"4[A|B]"]]]]]]
[:map ["f" [:map ["p" [:= "777"]]]]]
[:map ["d" [:map ["e" [:= "iia"]]]]]
[:map ["a" [:map ["b" [:= "hhhj"]]]]]
[:not [:or [:map ["d" [:map ["e" [:re #"84[B]?$"]]]]] [:map ["a" [:map ["b" [:= "HHJE"]]]]]]])
This works very nice for validation where I can just surround the whole thing with an [:and ...] and it will validate any data.
However the real value would come if I where able to generate data from this list of specs since that is not available for the legacy system. However generating data will not work when surrounding with [:and ...] since each [:map] only describe on key.
What I think I need to do is merge all the sub specs into a single [:map ...] spec, however I would like to know
if this is possible and where to start with such and algorithm. Has any work like this been explored using Malli before? I see there is a union operator but that is not what I need.
The above example show the full extension what the grammar contains which is basically :not :and :or :map := :re.
Any input for this problem would be very much appreciated.#2024-02-2408:31Henrik LarssonI guess if I replace every :or with :union that would take me one step closer, however what should I do with the :and?#2024-02-2408:36Henrik LarssonI think what I will try is to compile into only :not :or implementing rewrite rules for :and then I should just be able to use :union to merge all rules into one I think.#2024-02-2411:16Henrik LarssonThis looks interesting https://github.com/bsless/malli-keys-relations any info why it got abandon?#2024-02-2418:04Henrik LarssonI have now solved it I think, what I do is a create a schema containing all values without any constraints, I use this schema to generate values, then I validate the results against to [:and ...] of all the expressions I have. Not sure if there is a smarter way but this looks to work very well, will need more testing before I know for sure.#2024-02-2517:04ikitommihi. so the problem is that with :and? it uses the first branch as generator and rest to validate the generated values. if the first one gives too wide results, the rest don’t easily match.#2024-02-2517:05ikitommiyou could:
1. put the most spesific first
2. use a custom generator at :and#2024-02-2517:07ikitommi(def schema1
[:not [:map ["a" [:map ["b" [:re #"dd[A|B|C|F|D|E]"]]]]]])
(def schema2
[:or
[:map ["c" [:map ["d" [:= "3"]]]]]
[:map ["c" [:map ["d" [:= "2"]]]]]
[:map ["c" [:map ["d" [:= "9"]]]]]
[:map ["c" [:map ["d" [:= "99"]]]]]])
(mg/generate [:and schema1 schema2])
;=> Couldn't satisfy such-that predicate after 100 tries.
(mg/generate [:and schema2 schema1])
;=> {"c" {"d" "3"}}
(mg/generate [:and {:gen/schema schema2} schema1 schema2])
;=> {"c" {"d" "2"}}
#2024-02-2517:08ikitommithe :not doesn’t generate very well:
(mg/sample schema1)
;(nil
; nil
; (#uuid"303ca3c6-7463-4be4-8f01-07a2b2606446")
; 2.0
; nil
; [:R?/L??]
; nil
; ({-5 -})
; #{0 -0.3046875 f/yA. true "ch[Kd4.." -8/3 :-/k_}
; #{[] #{-5}})#2024-02-2517:08ikitommihope this helps#2024-03-0709:17Henrik LarssonThanks for this, it makes sense that the first and works as a generator, that explains why what I did actually works.#2024-02-2318:48frozenlockIs it possible to 'query' Malli schemas? For example, "give me all the fields that can accept a collection of integers"#2024-02-2407:31steveb8nOne solution would be to combine specter and deref-recursive. An interesting idea would be deref-recursive into a data script db, then datalog#2024-02-2517:09ikitommicould you @U0ERZQ1K2 preudocode what you would like to see? e.g. input + output. I would just m/walk over the schema…#2024-02-2616:46frozenlockYes... I'll come back on this with a usable example, I need to convert my stuff from spec/spec-tools first.#2024-02-2610:15Bingen Galartza IparragirreI think I spotted a bug with strip-extra-keys-transformer and map-of:
dev> (malli.core/encode [:map [:a [:string]] [:b [:string]]] {:a "a" :b "b" :c "c"} malli.transform/strip-extra-keys-transformer)
{:a "a", :b "b"}
dev> (malli.core/encode [:map [:a [:string]] [:b [:string]]] {:a "a" :c "c"} malli.transform/strip-extra-keys-transformer)
{:a "a"}
dev> (malli.core/encode [:map-of [:string] [:map [:a [:string]] [:b [:string]]]] {"string" {:a "a" :b "b" :c "c"}} malli.transform/strip-extra-keys-transformer)
{"string" {:a "a", :b "b"}}
dev> (malli.core/encode [:map-of [:string] [:map [:a [:string]] [:b [:string]]]] {"string" {:a "a" :c "c"}} malli.transform/strip-extra-keys-transformer)
{}
The last example should not be empty#2024-02-2610:15Bingen Galartza IparragirreI found a comment in [this PR](https://github.com/metosin/malli/pull/963)
There is a pending issue when an invalid key in :map-of instead of being removed from the value invalidates the whole :map-of. This would probably require recursive validation, which may add some computational complexity.
#2024-02-2610:16Bingen Galartza IparragirreBut I can't find an open issue :thinking_face: Should I open one? or it's already somewhere?#2024-03-0606:37ikitommiplease open#2024-02-2616:52frozenlockIs this the correct way to say "I want a vector with a minimum length of 2, and the first item must be a keyword"?
(m/validate [:and [:cat {:min-count 2} :keyword [:* :any]] vector?] [:kw "a" 1])
; => true
Using this with a generator fails after 100 tries...
(mg/generate [:and [:cat {:min-count 2} :keyword [:* :any]] vector?])
Execution error (ExceptionInfo) at clojure.test.check.generators/fn (generators.cljc:435).
Couldn't satisfy such-that predicate after 100 tries.
#2024-02-2617:56Stig BrautasetI think you want a tuple? Something like this:
[:tuple :keyword [:* :any]]
#2024-02-2617:57Stig BrautasetAlthough that https://malli.io/?value=%5B%3Akw%20%22a%22%201%5D&schema=%5B%3Atuple%20%3Akeyword%20%5B%3A*%20%3Aany%5D%5D, hmm...#2024-02-2617:58Stig BrautasetAh, because you don't want a tuple you want a cat:
[:cat :keyword [:* :any]]
(Better yet:
[:cat :keyword [:+ :any]]
to ensure there are at least 2 items)#2024-02-2617:59Stig BrautasetSee https://malli.io/?value=%5B%3Akw%20%22a%22%201%5D&schema=%5B%3Acat%20%3Akeyword%20%5B%3A*%20%3Aany%5D%5D#2024-02-2617:59Noah Bogartthat's what frozenlock has originally#2024-02-2617:59Noah Bogarti think it's an error in the generator#2024-02-2618:03Stig BrautasetI would drop the {:min-count 2} and replace :* with :+ which might improve the generator.#2024-02-2618:04Stig Brautaset[:and [:cat :keyword [:+ :any]] vector?]
#2024-02-2618:47frozenlockBut that still doesn't work with generators, right?
(mg/generate [:and [:cat :keyword [:+ :any]] vector?])
Execution error (ExceptionInfo) at clojure.test.check.generators/fn (generators.cljc:435).
Couldn't satisfy such-that predicate after 100 tries.
#2024-02-2618:48Stig BrautasetI don't understand why it does not work with generators :thinking_face:
It works in the http://malli.io page 🤷#2024-02-2618:58frozenlockOh wait, it looks like it can work if vector? is first:
(mg/generate [:and vector? [:cat :keyword [:+ :any]]])
=> [:!P4+
-1708474864121283778364080570989949433939248745244365942N
4.0
true
##NaN
"L`{]aY:;Vj9:4'/i#G1=E!!g~r"
#uuid "3231ba6a-b098-41d7-b9ce-8fc700d1d67a"
false
0.0019298376955703134
-
:Z910dDG9.
YbA+dM/J*.._CI8]#2024-02-2618:59Noah Bogartmaybe it keeps generating conforming sequences for the :cat and then those aren't vectors so it fails#2024-02-2819:41ambrosebsThe key thing to understand is that malli generates values from the first child of an :and, and then only picks ones that pass the entire schema.
So the :cat has to go first to have a reliable generator.
This should work: [:and [:cat {:gen/fmap vec} :keyword [:* :any]] vector?]
https://malli.io/?value=(%3Aa)&schema=%5B%3Aand%20%5B%3Acat%20%7B%3Agen%2Ffmap%20vec%7D%20%20%3Akeyword%20%5B%3A*%20%3Aany%5D%5D%20vector%3F%5D#2024-02-2820:53ambrosebsAdded this tip to the readme, if anyone has time to proof read: https://github.com/metosin/malli/pull/1012#2024-02-2620:26frozenlockLooks like the parser is losing the metadata along the way. Bug?
(def id-schema
[:and
;; Nested :and to be able to add metadata to generated content
[:and {:gen/fmap #(with-meta % {:kw-id true})}
[:cat {:gen/fmap vec}
:keyword
[:+ :string]]
vector?]
[:fn #(:kw-id (meta %))] ;
If you comment out the metadata test and try again, it works:
(def id-schema
[:and
;; Nested :and to be able to add metadata to generated content
[:and {:gen/fmap #(with-meta % {:kw-id true})}
[:cat {:gen/fmap vec}
:keyword
[:+ :string]]
vector?]
;[:fn #(:kw-id (meta %))] ; #2024-03-0606:40ikitommiplease write an issue out of this#2024-02-2808:06Tommi MartinHello what would be your general tips to making schemas with malli?
I've successfully made a few map schemas but i'm struggling to use them effectively as I get a "invalid-type" validation errors with large schema files and large values inserted to those. Making it difficult and slow to actually track down what went wrong. Adding a custom error to these only drops the entire schema and entire value into the error, nothing mory.
Do you have any specific tips on how to structure the schema to get as accurate error messages as possible? My current schema is a map contain vectors, containing maps up to 5 levels. It's all defined as a single entity with the vector syntax alone, no registry.#2024-02-2819:21ambrosebsFWIW it's not just you, this is an area for growth for malli. Some ideas with that in mind:
You might want to https://github.com/metosin/malli/blob/875caae5ecbeecbe32d40af53c0732bec717f70a/src/malli/error.cljc#L29 the default invalid-type error message to be more specific.
e.g., https://github.com/fluree/db/commit/4ff41ae12fe5588b0e64da93ff45ea14d72499cc#diff-5b7e602fc7bc937648bce72499d1c9c24ead635bf2e72468c224769b43f9263aR212-R219
::m/invalid-type
{:error/fn (fn [{:keys [schema value]} _]
(if-let [expected-type (-> schema m/type)]
(str "should be a " (case expected-type
(:map-of :map) "map"
(:cat :catn :sequential) "sequence"
:else (name type)))
(str "type of " (pr-str value) " does not match expected type")))}
Use :multi instead of :or.#2024-03-0606:48ikitommiagree, the default error is bad, should be easy to make better as Ambrose pointed out 🙂#2024-02-2809:52SardtokI just wanted to make sure I'm not doing something stupid. Is there a simpler way to require that fields are set, but only if one specific field is set to true?
[:multi {:dispatch :x}
[true [:map [:x :boolean]
[:y :int]]]
[::m/default [:map [:x {:optional true} [:maybe :boolean]]
[:y {:optional true} [:maybe :int]]]]]#2024-02-2818:28ambrosebsI'd guess this is the most performant, might not be the simplest.#2024-02-2818:28ambrosebsthe alternative would be something like [:or big-map another-big-map] and a lot of time would be spent figuring out which schema to use.#2024-02-2818:40SardtokUsing :or gave worse results for the errors with explain (and reitit's coercer). It would generate errors for both branches, which in the case of the coercer merged into a single set of humanized errors without the full path identifying the branch.#2024-02-2818:42ambrosebsYes, good point. :or is not good for error messages.#2024-03-0606:50ikitommideclarative key/value relations are not simple, here is one try: https://github.com/bsless/malli-keys-relations#2024-02-2817:12vemv> (malli.generator/generate [:map [:success? {:gen/return true} boolean?]])
{:success? false}
Looks like :gen/return isn't doing what I want to here - suggestions?#2024-02-2818:24ambrosebsPerhaps try pushing it in? [boolean? {:gen/return true}]#2024-02-2818:26ambrosebshmm but there's a test case for this syntax in the malli test suite. I'm not sure.#2024-02-2818:28vemvHey there 👋
I had tried this variation as well, inspired by the readme
> (malli.generator/generate [:map [:success? [:and {:gen/return true} boolean?]]])
{:success? false}#2024-02-2818:30ambrosebsHey!
Curious, does :gen/elements [true] work?#2024-02-2818:31vemvCertainly does, thanks!#2024-02-2818:32ambrosebsAlright! Looks like :gen/return is a newer feature, maybe it's missing something.#2024-02-2818:32vemvUnrelated, lately I've been really wanting a Schema -style zero-cost defprotocol for Malli. Maybe one day I'll find the time.#2024-02-2818:35ambrosebsYes, I think I told you that I was working on a common library extracting out the implementation details of s/defprotocol.
I got stuck, just made it public, maybe you'll find it inspiring? https://github.com/frenchy64/instrument-defprotocol#2024-02-2818:36vemvoh, interesting :)
> I got stuck
What's the lib state?#2024-02-2818:40ambrosebsPretty rough, unreleased, but it at least separates out the main ideas of what's needed to instrument protocols visually. I think if you read https://github.com/frenchy64/instrument-defprotocol/blob/main/src/com/ambrosebs/instrument_defprotocol/clj.clj
file you should get a good idea of what's involved.
IIRC I found it difficult to decide how leaky I wanted to make the abstraction, because there were performance benefits for being more flexible but also potentially it made it easier to shoot yourself in the foot.#2024-02-2818:42vemvDid the Schema impl play out nicely long term? Perhaps I'd be better off with a non-generic solution#2024-02-2818:49ambrosebsEither no-one used it or it worked perfectly 🙂#2024-02-2818:51ambrosebsWorks on the latest Clojure master based on the cron CI. Not sure if I test the latest CLJS though.#2024-02-2818:53vemvThanks! Hope I can hack something with it this year. Should be pretty low-risk as one can always replace custom/defprotocol with defprotocol - nothing else should be adapted#2024-02-2818:54ambrosebsBut I agree, a non-generic solution is probably best.#2024-02-2818:54ambrosebsAwesome!#2024-02-2819:02radhika@U45T93RA6 hm, which version of malli were you using with :gen/return?#2024-02-2819:05vemv[metosin/reitit-malli "0.5.18" :exclusions [[org.clojure/tools.reader] [org.clojure/core.rrb-vector]]]
[metosin/malli "0.8.2"]
Didn't realise this project had such an old version :) that tag is from Feb 14, 2022
Will update + retry, thanks!#2024-02-2819:10vemvThat was it 🤝#2024-02-2821:09vemvCan I use malli.util/transform-entries or the like to transform the case of my schema keys? e.g. apply ->kebab-case over each key#2024-03-0414:16respatializedIs there an easy way to select only the entries of a :map schema that have required keys?#2024-03-0606:34ikitommisomething like:
(->> [:map
[:a :any]
[:b {:optional true} :any]
[:c :any]]
(m/children)
(filter (comp not :optional second))
(map first))
; => (:a :c)#2024-03-0606:35ikitommifor EntrySchemas, children is guaranteed to have a tuples of [key props value]#2024-03-0516:01lambdamHello everyone,
We're introducing malli on a project that started with spec. We'd like to get rid of the spec definitions bound to namespaces, since namespaces are organized as a tree and our information is structured as a graph (i.e. non hierarchical).
We're defining a big flat repository and as long as we're using pure data, the order of definitions doesn't matter.
I started to use some functions of the malli.util namespace (mainly malli.util/merge) for some precise composition of schema and I get :malli.core/invalid-schema errors on project startup. The schema is actually defined but maybe after in the namespace hierarchical dependencies. When I have a live REPL, I don't get this error when refreshing the project with tools.namespace. And when I remove the use of malli.util functions, the project starts without errors.
My question is: do functions of the malli.util namespace check that schemas exist in the default registry before manipulating the data structures that describe schemas?
Thanks#2024-03-0607:01ikitommiyou can make some schemas lazy, e.g. :map and :multi can be configured to work so. But by default, most schemas are eager. Merge etc. should be lazy too :thinking_face:#2024-03-0612:21lambdamJust to give the context, we're defining an extended default registry like so (fake code but real structure):
(ns schema.registry
(:require schema.db
schema.user
schema.order))
(let [my-registry (mr/composite-registry
(mr/fast-registry (m/default-schemas))
(mr/var-registry)
;; ☝️ taken from malli.core
;; 👇 our schema definitions as flat maps
schema.db/registry
schema.user/registry
schema.order/registry)]
(mr/set-default-registry! my-registry))
;; malli.core aliases so that we ensure that registry is defined
;; before any usage of these functions
#?(:clj
(defmacro => [given-sym value]
`(m/=> ~given-sym ~value)))
(def coerce m/coerce)
(def explain m/explain)
(def validate m/validate)
(ns schema.db)
(def registry
{:db/id :int
:db/ref :db/id
:db/refs [:sequential :db/ref]})
(ns schema.user)
(def registry
{:user/id :db/id
:user/name :string
:user/email :string
:user/orders :db/refs})
(ns schema.order)
(def registry
{:order/id :db/id
:order/user :db/ref})
Here we have a (silly but) valid circular reference between users and orders.
In some namespaces, the usage of malli.util functions would be very convenient (`merge` or optional-keys in our case). But since all those namespaces are loaded before the new default registry, having eager function (if eager = use of deref) would be impossible.
This is weird in a way since we're manipulating pure data to describe schemas.
It would be convenient to have the possibility to use malli.util functions used in a lazy way and the cross usage of schemas checked at creation time. This is what Datomic does in transactions where references with tempids (~ the graph of information) can be described with a sequence a flat maps.
So, as of today:
1. Is it possible to call malli.util functions in a lazy way (without derefs) with the option arity? We quickly looked for it but didn't find anything.
2. Is it possible to have the integrity of existing schemas defined in a registry checked at creation time (specifically schemas used in :map schemas: e.g. [:map :user/name :user/email ...] )?
3. Is this approach common or do people tend to use the default registry with locally defined schemas (e.g. [:map [:user/name :string] ...])?
Thanks#2024-03-0606:11ambrosebsIf you've ever wanted more powerful and expressive specs, you might enjoy my live stream tomorrow. Relevant to schema/spec/malli users.#2024-03-0614:41lambdamThe subject looks fantastic!
Sadly, the live presentation will at 1:30 in the morning here in Paris, France.
I'll watch the replay.#2024-03-0717:48ambrosebsTrimmed video is here https://www.youtube.com/watch?v=QE3PZLlefUE#2024-03-0803:21escherizeThanks for posting this! I was about to post it here now.#2024-03-0803:24escherizeI am still watching intently, but I think, at around the 21 minute mark, I think you might have skipped over https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-guards#2024-03-0804:53ambrosebs@U051GFP2V I'm aware of them, they're only really useful for non-higher-order functions. here's the identity specs I tested https://github.com/typedclojure/typedclojure/blob/main/typed/clj.spec/test/typed_test/clj/spec/identity.clj#2024-03-0804:53ambrosebsBut check out what it looks like to achieve something similar for comp https://github.com/typedclojure/typedclojure/blob/7ef50ae3db663a713960a06b804ca0927da71a92/typed/clj.spec/test/typed_test/clj/spec/comp.clj#L14-L27#2024-03-0815:41escherizeYeah, I should have mentioned once you got into the typeclass sort of stuff (typing comp escpecially), I can see how your work is a different level of abstraction. For more specific types it is possible, in Malli, to have a relation from inputs to outputs using function schemas, but it isn’t derivable from the specs themselves.#2024-03-0816:22ambrosebsThe relationships between inputs and outputs was a bit of a sleight of hand for presentational purposes. If I were to give the presentation to you, it would be very simple:
1. insight 1: a type variable can be replaced by an arbitrarily specific type and the type will still be inhabited by the same program (e.g., identity is both X->X can be 1->1)
2. insight 2: you can get a surprisingly accurate generative test by making specs that look like the instantiations (like 1->1, (1->2)[1]->[2])
3. You can only write only spec per function, so let's make a syntax that works both for instrumentation and excercising a function that uses this trick "for free" instead of worrying about :fn and even surpassing it in some cases).#2024-03-0816:25ambrosebsI didn't mention in the talk that you can get back the instrumentation spec in many cases by instantiating variables to :any.#2024-03-0816:31ambrosebsAnother thing to understand about the talk was I challenged myself to make it understandable to a 10 year old who hasn't programmed yet. 🙂#2024-03-0816:32ambrosebsI'm very proud to say he summarized my talk as: "specs must be specific to be useful".#2024-03-0816:32escherizeI appreciated the starting-slow style. Leaving off namespace aliases etc, makes things a lot clearer.#2024-03-0816:32ambrosebsTrue, the syntax is really difficult for slides.#2024-03-1618:12vemvWatched - nice!
Curious, what's are your thoughts on Spec function instrumentation :fn s? (or Malli https://github.com/metosin/malli/blob/fb467791f3748f8eb2795bbcc029e583c028875e/docs/function-schemas.md#function-guards)
Frankly, I've very rarely wanted to reach for them. Although I'd be open that the idea of them (or their leveled-up counterparts) bringing in some power.#2024-03-1620:27ambrosebs@U45T93RA6 thanks! about :fn I guess it's an escape hatch for dependent schemas. My main reservation is that it's code, not a spec, so there's less you can do with it besides just run the code.
I tried to push :fn about as far as you can go with generative testing. https://github.com/typedclojure/typedclojure/blob/7ef50ae3db663a713960a06b804ca0927da71a92/typed/clj.spec/test/typed_test/clj/spec/comp.clj#L14-L27#2024-03-1620:31ambrosebsBut I think :fn fits with the rest of spec's design, which is mostly about hardening functions that pass domain-specific data around (and not super concerned with higher-order implementation-level things).#2024-03-1620:46ambrosebsI don't have any concrete ideas on how to improve :fn but the spirit of what I hope is possible is more like:
(fspec :args (dcat :a int? {:keys [a]} :b (coll-of any? :min-count a)))
"dependent cat" where you can bind the results of conforming the args as you go and then can use it to tighten the specs on the right. But this doesn't really work AFAICT.#2024-03-1620:48ambrosebse.g., raises too many questions about nesting with sequence regex specs, and probably only works left-to-right. But maybe it's a little more useful to programmatically inspect?#2024-03-1620:50ambrosebsBut the main downside to :fn that comes to mind in practice is that it doesn't participate in generating arguments for the function. You just blindly generate the args, and then hope it conforms to the :fn later on https://github.com/clojure/spec.alpha/blob/c630a0b8f1f47275e1a476dcdf77507316bad5bc/src/main/clojure/clojure/spec/test/alpha.clj#L410#2024-03-1620:56vemvThanks for the overview :)
Would be curious to see where all this work goes. Is it ambitious nowadays, or something more research-y / side-projecty?#2024-03-1621:53ambrosebsI think it's pretty grounded, I have prototypes for spec/malli/schema. Working on malli right now, seems the most flexible. But really a lack of community interest is the main roadblock. The responses to the talk have been encouraging.#2024-03-1621:55ambrosebsWell, the other roadblock is perfectionism. Or at least, wanting to get it right so I don't need to make breaking changes. 🙂
For example, I want to write specs that work for all arities of map/comp/every-pred etc. It's very fun, but it's hard to tell whether it needs more hammock time.#2024-03-1622:05ambrosebsIt's been about 3-4 years, maybe it's enough time? 😉
The question I'm trying to answer right now is about the "kind" of specs (like kinds in type theory, or if you're not familiar, the question of "what is the spec for a spec").
So far I think regular non-regex specs are of kind "Spec". This is so you can write (all [x :- Spec] [:=> (cat x) x]) and disallow instantiating x with [:cat now identity takes fives args] .
So the next question is, what's the kind of regex specs? I think you can just lift them to the kind level. like the spec for memoize is (all [x :- (* Spec), y] [:=> [:=> x... y] [:=> x... y]]) . That's what I'm wrestling with atm.#2024-03-1622:11ambrosebsOne of the joys in this work is that spec/malli/schema have given me a playground to work out these ideas. The same problems pop up in Typed Clojure, but it's so much more difficult to even formulate the problem. My hope is that this will teach me a bit about how to support proper sequence regex types in Typed Clojure.#2024-03-1622:14ambrosebsBut the other joy is that I've been really impressed with the utility of these extensions to spec/malli/schema, especially compared the implementation effort vs implementing a type system. It's got me reconsidering what a "gradual type system" might look like.
Traditionally, it's static typing protected by instrumentation at the typed-untyped boundary. But, what if it was static typing protected by generative testing?#2024-03-1622:16ambrosebsConcretely, perhaps if you annotate a type as "no-check" in Typed Clojure (don't statically check its definition), it instead runs generative tests at type-checking time?#2024-03-1622:18ambrosebsOr even, apply that same idea to clj-kondo. Enhance clj-kondo's type syntax to include polymorphism, but verify it with generative testing instead. Then just use the simplified type during clj-kondo static checking.#2024-03-1622:22escherizeWow thanks for writing up your thoughts: novel and interesting!#2024-03-1622:23vemv> But really a lack of community interest is the main roadblock.
Being optimistic, it might be simply some sort of chicken-and-egg problem, i.e. in absence of incredibly attractive features, adoption will be smaller, which in turn means less development.
In my perspective, most people think in terms of 'features'. They might care less about the theory behind as long as there's some big ROI.
Beside from good old type checking, I tend to want, for instance, whole-system generative testing (e.g. my distributed system behaves nicely even if X times out and Y fails). Results (as opposed to exceptions) seem a good fit.
I wonder if, should I give that sort of task a shot, would 'naive specs' cut it, or would I end up needing higher level constructs 😄#2024-03-1622:38ambrosebsBeing optimistic, it might be simply some sort of chicken-and-egg problemYeah you hit the nail on the head. I haven't gotten to ride the adoption/interest wave yet to a stable release, which is both a blessing and curse.
> I wonder if, should I give that sort of task a shot, would 'naive specs' cut it, or would I end up needing higher level constructs
I think if it you needed high-level constructs to test systems end-to-end, they would already be supported by spec given how pragmatic it is.
This work is definitely more niche. I tried to lay it out explicitly in my talk. It's hard to come up with relevant features like "this is how to fuzz test your transducer" because most people don't (get paid to) write transducers.#2024-03-1622:40ambrosebsDevelopers of transducer libraries might be confident enough to not want to fuzz test them. I'm not exactly sure how to market these things.#2024-03-1622:41ambrosebsPerhaps if the entire vision is realised of static checking + generative testing, then things will be more compelling.#2024-03-1622:43vemv> I tried to lay it out explicitly in my talk.
Yeah I had gotten the impression that this stuff shines the most when writing/hacking clojure.core or so.
At the same time I'm open to be surprised as for broader applications
> Perhaps if the entire vision is realised of static checking + generative testing, then things will be more compelling.
Sounding great, I'd be curious to see a clj-kondo + higher-level + generative thing, sounds like a good composition#2024-03-1622:44ambrosebsMaybe even a pitch like "well, we all have a utils ns right? here's how to fuzz test that ns".#2024-03-1622:45ambrosebs> Yeah I had gotten the impression that this stuff shines the most when writing/hacking clojure.core or so.
Yes. And I guess utils namespaces look the most like clojure.core in most code bases. Maybe that's a good angle.#2024-03-0817:26Henrik LarssonHow can I construct a set spec that assert the present and absence of certain values. I am generating specs dynamically based on parsed expressions so I would need a dynamic way to construct my spec. What I am looking for is something like.
(mg/generate
[:and
[:set [:enum 1 2 3 4 5 6 7 8 9]] ; This is the full domain of the set, all combinations should be generated and then filtered based on predicated as described below
; ex1 set should include 1 AND 4 but NOT 8 other values are ok
; ex2 set should include 1 AND 4 but NOT 8 other values NOT ok
; ex3 set should include 3 OR 4 but not both
])
#2024-03-0817:31Noah Bogartmaybe this should be :or schemas, a union of different conforming values#2024-03-0817:44Henrik LarssonNo each rule I have now will restrict the set in some way and all the restrictions need to be valid for the set to be valid. I have the same setup for a map and it works great, however I can not find any good example how to restrict a set/collection using Malli.
Or maybe I misunderstand what you mean, can you give some example code that would generate what I want using :or?#2024-03-0817:58escherizeFor the example you gave, you use :fn schemas to invalidate certain shapes#2024-03-0818:00escherizeI filled in ex3 here:
(mc/validate
[:and
;; This is the full domain of the set, all combinations should be generated and then filtered based on predicated as described below
[:set [:enum 1 2 3 4 5 6 7 8 9]]
;; ex1 set should include 1 AND 4 but NOT 8 other values are ok
;; ex2 set should include 1 AND 4 but NOT 8 other values NOT ok
;; ex3 set should include 3 OR 4 but not both
[:fn (fn [s] (not (and (contains? s 3) (contains? s 4))))]
]
#{3 4})
;; => false#2024-03-0818:09Henrik LarssonIs there any way to force more then 100 tries when you generate values, looking at the code I get the feeling that this value is not meant to be changed?#2024-03-0818:19escherizefor generating tricky datastructures, there are some tools, the simplest is supplying elements with :gen/elements (these all go on the and), or :gen/fmap which would be a function that can return valid sets for you#2024-03-0818:20escherizeI think :gen/gen can be a generator. hangon I’ll find the docs#2024-03-0818:20escherizehttps://github.com/metosin/malli?tab=readme-ov-file#value-generation#2024-03-0818:45Henrik LarssonThe simplest would be just to be able to generate more values, the shape is not that tricky and all combinations would probably be generated for 5000 tries. So would be nice to just be able to increase the number of tries.
Currently I do this by not wrapping the first clause in an and, instead I just generate a value and then filter if over the validation of the remaining schema validations. Then I can loop this until I find a match. Since I generate the schema dynamically it is nice to not have not to custom schemas. I am trying to keep them simple as possible and then rather just spend some more time on random generation.#2024-03-0818:57Henrik LarssonAs I understanding if I am to do what you say I would need to dynamically build but generators rather then schemas?#2024-03-0907:38ikitommiBuilding a generator dynamically too would make sense here.#2024-03-0907:41ikitommiIf someone has a good proposal for a generator combinator thing in malli, happy to hear. E.g. where :and would collect generator part from it's children and construct a functioning generator out of that automatically.#2024-03-1508:26Henrik LarssonI am now using m/generator to get the generator then I can use test.check instead. Combining that with malli specs has really been a nice fit for my use case so far.#2024-03-0920:20Pavel MugyrinHi everyone! Is it possible to make transformations work in a depth-first fashion? Here is the example:
(def TimestampAndMap
[:map {:encode/edn pr-str
:decode/edn clojure.edn/read-string}
[:timestamp [:time/instant {:encode/timestamp #(.getEpochSecond %)
:decode/timestamp #(Instant/ofEpochSecond %)}]]])
=> (m/encode TimestampAndMap {:timestamp (Instant/now)} (mt/transformer {:name :timestamp} {:name :edn}))
"{:timestamp #object[java.time.Instant 0x8e217f2 \"2024-03-09T20:15:38.302046177Z\"]}"
;; but i would rather want to see
"{:timestamp 1710015535}"#2024-03-1318:37jmvis it possible to add descriptions to the values in an enum schema?#2024-03-1409:22Adham OmranAs in a description per value?#2024-03-1415:40jmvyeah, basically to document the meaning of different values in an enum#2024-03-1501:01David GHi, how do you serialize function schemas? Or in general, how to serialize the explain data?
For example:
(m/explain [:fn pos?] 0)
;; => {:schema [:fn #function[clojure.core/pos?]],
;; :value 0,
;; :errors
;; ({:path [], :in [], :schema [:fn #function[clojure.core/pos?]], :value 0})}
Or with [:multi {:dispatch :type} [:a :int] [:b :string]] the Cider REPL evaluates it correctly but :schema at top level or within the errors is a function:
(mc/explain [:multi {:dispatch :type}
[:a :int]
[:b :string]]
"string")
I tried using m/ast but stumbled upon :fn and that didn't serialize.
Any alternatives? I could name all my schemas or function schemas#2024-03-1503:10ambrosebsSome alternatives I'm aware of:
user=> (m/explain [:fn #'even?] nil)
{:schema [:fn #'clojure.core/even?], :value nil, :errors ({:path [], :in [], :schema [:fn #'clojure.core/even?], :value nil, :type nil})}
user=> (m/explain [:fn '(fn [x] (even? x))] nil)
{:schema [:fn (fn [x] (even? x))], :value nil, :errors ({:path [], :in [], :schema [:fn (fn [x] (even? x))], :value nil, :type :sci/error})}
user=> (m/explain [:fn `(fn [x#] (even? x#))] nil)
{:schema [:fn (clojure.core/fn [x__11833__auto__] (clojure.core/even? x__11833__auto__))], :value nil, :errors ({:path [], :in [], :schema [:fn (clojure.core/fn [x__11833__auto__] (clojure.core/even? x__11833__auto__))], :value nil, :type :sci/error})}
user=> (m/explain [:fn `#(even? %)] nil)
{:schema [:fn (fn* [p1__11870__11871__auto__] (clojure.core/even? p1__11870__11871__auto__))], :value nil, :errors ({:path [], :in [], :schema [:fn (fn* [p1__11870__11871__auto__] (clojure.core/even? p1__11870__11871__auto__))], :value nil, :type :sci/error})}#2024-03-1504:28Ben SlessThat's one reason I dislike function schemas. Everything becomes a hassle#2024-03-1507:43Sagar VrajalalIs there a way to force a particular selection inside an :or using a property?
Alternatively, I could transform
[:map
[:value
[:or
[:map [:date :time/local-date]]
[:map [:text :string]]
[:map [:number :int]]]]]
into
[:map
[:value
[:map [:date :time/local-date]]]]
But it doesn't seem so straightforward if the only information I have during runtime is the keyword :date
I could obviously compose the schema programmatically during runtime but I want to be able to generate values at the REPL too, so I'd like to keep my schema defined as-is and just modify it at runtime#2024-03-1515:58ambrosebsCould use :gen/schema on the :or to override the generator spec for a spec.#2024-03-1508:29Henrik LarssonWhat are the limits and rules surrounding the explain functionality?
For example the following returns unknown error
(me/humanize
(m/explain [:and
[:not
[:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["a" [:map ["b" [:= "2"]]]]]]]]
{"a" {"b" "1"}}))
The error is simple to understand by a human but somehow explain do not understand it. My spec is generated dynamically so just saying I should change how my spec is written will not help. I need to understand why this fails so I know how I should rewrite the dynamic generators to actually support explain.#2024-03-1509:02Henrik LarssonThis works
(me/humanize
(m/explain [:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["a" [:map ["b" [:= "2"]]]]]]
{"a" {"b" "3"}}))
So I guess it is the negation that causes the unknown error#2024-03-1516:07escherizeprotip: :or schemas give worse error messages than :multi schemas, if you can transform your code to use :multi, they perform better too, since they don’t have to check every branch (they’d report an error for each :or branch).#2024-03-1516:14ambrosebs(humanize
(m/explain [:multi {:dispatch map?}
[true [:multi {:dispatch #(contains? % "a")}
[true [:map ["a" [:multi {:dispatch #(contains? % "b")}
[true [:map ["b" [:not= "1"]]]]
[false :any]]]]]
[false :any]]]
[false :any]]
{"a" {"b" "1"}}))
{"a" {"b" ["should not be 1"]}}
#2024-03-1516:18ambrosebsAs you noticed, :not erases any error messages, so you need to formulate the spec positively up to the leaves. Negating a nested map is a very broad spec. That's equivalent to [:not [:map ["a" [:map ["b" [:= "1"]]]]].#2024-03-1609:20Henrik LarssonI trying to understand how big the task would be to rewrite what I have to use :multi as in your example. What I do not is that I have lots of boolean expressions I parse from strings getting an AST that maps very close to exactly what I have in malli which is nice, then I combine then inside a :and spec in malli. This works nice for my use case of generating data (with some extra magic), however what I am trying now is to use it to also validate data and get good error messages. The expressions I build can get big and complex but they will always have forms like
[:and
[:not [:map ["a" [:map ["b" [:= "1"]]]]]]
[:map ["a" [:map ["b" [:= "1"]]]]]
[:not
[:or
[:map ["c" [:map ["d" [:= "1"]]]]]
[:map ["c" [:map ["d" [:= "2"]]]]]]]
[:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["c" [:map ["d" [:= "2"]]]]]]]
This is 4 rules combined inside an :and.
Any pointers to get me started in the right direction for using :muti for expressions like this, How would I express :and. To move :not I guess I would need to use De Morgans Law and rewrite my expressions.
And an expression like
[:not
[:or
[:map ["c" [:map ["d" [:= "1"]]]]]
[:map ["c" [:map ["d" [:= "2"]]]]]]]
dont look to have a straight forward translation to multi. Rather I guess I should try to trans form that to
[:map ["c" [:map ["d" [:or [:not= "1"] [:not= "2"]]]]]]
#2024-03-1616:11ambrosebsI looked a bit more into this and I haven't worked out how to completely eliminate :or. The problem is that it's not obvious what to use as the dispatch function in general. I'm going to think about it over the weekend and get back to you.#2024-03-1616:26ambrosebsPerhaps one solution would be to push the :not's in to leaves, and allow :or but improve the error message calculation logic for :or. I'll look into that too.#2024-03-1616:39ambrosebsHere's my progress on pushing in :not . Please try it out. Instead of generating [:not s] instead return the result of (malli.negate/negate s). https://github.com/frenchy64/malli/compare/master...fecb20a03d3165816026a47bd10bd8bc488f2647#2024-03-1616:48ambrosebsOh, but you said that [:or :map :map] worked right? So perhaps this will work for you^#2024-03-1619:20Henrik LarssonHawing the following still return unknown error
(me/humanize
(m/explain
(mneg/negate
[:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["a" [:map ["b" [:= "2"]]]]]]
){"a" {"b" "1"}})) #2024-03-1619:24Henrik LarssonI wonder if not using De Morgans Law and rewrite my expression would work the best, will experiment more with that. The the case
[:not
[:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["a" [:map ["b" [:= "2"]]]]]]]
Would then be rewritten as
[:and
[:map ["a" [:map ["b" [:not= "1"]]]]]
[:map ["a" [:map ["b" [:not= "2"]]]]]]
Which do give me the result I want
(me/humanize
(m/explain [:and
[:map ["a" [:map ["b" [:not= "1"]]]]]
[:map ["a" [:map ["b" [:not= "2"]]]]]]
{"a" {"b" "1"}}))#2024-03-1619:40Henrik LarssonWill have to do some full testing for more cases once I get back to work, but something like this is what im aiming at maybe
(defn negate-eq
[tree]
(w/postwalk
(fn [s]
(if (= s :=)
:not= s)) tree)
)
(w/postwalk
(fn [s]
(if (vector? s)
(case (first s)
:not
(case (first (second s))
:map (negate-eq s)
:or (into [:and] (mapv negate-eq (drop 1 (second s))))
:and (into [:or] (mapv negate-eq (drop 1 (second s))))
s) s)
s))
[:not
[:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["a" [:map ["b" [:= "2"]]]]]]])#2024-03-1620:20ambrosebsI think it's more subtle that than. Those aren't equivalent schemas.
(m/validate
[:not
[:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["a" [:map ["b" [:= "2"]]]]]]]
{})
;=> true
(m/validate
[:and
[:map ["a" [:map ["b" [:not= "1"]]]]]
[:map ["a" [:map ["b" [:not= "2"]]]]]]
{})
;=> false
#2024-03-1620:37ambrosebsFollowing my nose to try and make equivalent schemas is how I ended up with these kinds of negations:
(mn/negate
[:or
[:map ["a" [:map ["b" [:= "1"]]]]]
[:map ["a" [:map ["b" [:= "2"]]]]]])
;=> [:and
; [:or
; [:not #'clojure.core/map?]
; [:map ["a" {:optional true} :never]]
; [:map ["a" [:or [:not #'clojure.core/map?] [:map ["b" {:optional true} :never]] [:map ["b" [:not= "1"]]]]]]]
; [:or
; [:not #'clojure.core/map?]
; [:map ["a" {:optional true} :never]]
; [:map ["a" [:or [:not #'clojure.core/map?] [:map ["b" {:optional true} :never]] [:map ["b" [:not= "2"]]]]]]]]
#2024-03-1620:53Henrik LarssonI will need to try and understand all this, I still dont understand why my expressions are not generating the same schema when I do think the boolean algebra I use is valid so the two expressions should be equal. But I am far from an expert. I will keep working on this with the help from your inputs.#2024-03-1621:58ambrosebsPerhaps it might help to think of specs as sets of values, and the negation of a spec as the set of all the other values.
[:map [:a [:= 1]] includes {:a 1}. its negation includes every other value, including non-maps.#2024-03-1508:55seanstromHey there 👋
I have a question about malli's simple type support, specifically whether supporting :float would be useful in the schema definitions? I wonder because I can see that I can use :double , and that both Clojure and CLJS seem to prefer double-precision floats, but it could be useful to describe only needing single-precision floats. Does anyone have an opinion to share or some context on why :float is not currently supported?#2024-03-1509:05Henrik LarssonMight not be what you are after but you can still use float? like this for example
(m/validate [:set float?] #{(float 3.3)})#2024-03-1509:18seanstromYup yup that would work, though it seems in malli float? is sometimes treated as a double too. Which is a little confusing to me, I wonder if :float is mostly an alias to :double or is meant to be distinct?#2024-03-1522:21EugenHow would I implement the additionalProperties behavior from json-schema in malli?
https://json-schema.org/understanding-json-schema/reference/object#additionalproperties
for additionalProperties: false with properties and patternProperties - this allows for any property in the properties list + any properties that match the pattern.
I guess my question resumes to:
How can I implement patternProperties with malli ?
Context: I am exploring a manual malli schema for (docker) compose spec https://github.com/compose-spec/compose-spec/blob/master/schema/compose-spec.json .
I tried https://github.com/metosin/malli/pull/915 but it closes the service map because of additionalProperties: false so I can't add anything to them .
The implementation in PR#915 does not hanndle proeprties and patternProperties IMO .
And I am too new to malli to consider working on json-schema conversion (yet) .#2024-03-1522:30EugenI think one way to handle this is to use content dependent schemas ?
https://github.com/metosin/malli?tab=readme-ov-file#content-dependent-simple-schema#2024-03-1606:54Eugenwhat I don't seem to find in malli is how to implement validation for keys by regular expressions.
"patternProperties": {"^x-": {}}
I am looking at
Class-based schemas, contains java.util.regex.Pattern & js/RegExp
#2024-03-1610:52jamesmintramSimple question, and I am sure it has been asked before - but I am not really sure what I would be looking for!
I have a schema defined, I would like to 'decode' this data to get a map of coerced data back. At the moment mali returns everything I give it, coercing the stuff that is described in the schema and leaving everything else.
What I would like to do, is drop everything that is not defined in the schema - a little bit like a select-keys operation on a map.#2024-03-1613:41Safe HammadYou can use malli.core/explicit-keys to pull the keys from the schema. So the following will "clean up" your data before you decode it:
(require '[malli.core :as m])
(select-keys value (m/explicit-keys schema))
#2024-04-0212:24Daniel Ben ItzhakHad a similar issue but with nested maps, this solved the issue:
(m/decode
schema
value
(mt/strip-extra-keys-transformer))
Adding for visibility#2024-03-1614:00Lari SaukkonenTrying out malli-based custom autocomplete with emacs and cider:#2024-03-1614:06Lari SaukkonenSchema:
(def Schema
[:map
[:field
[:map
[:id :int]
[:another-field
[:vector
[:map
[:id :int]
[:status [:enum :WORKING :NOT_WORKING :OLD]]
[:id :int]]]]
[:imaginary-subfield
[:map
[:field-first :string]
[:field-second :string]]]]]])
It has custom compliment source that reads prefix and form to match it with the schema and offers it through cider-nrepl middleware that i changed a bit.#2024-03-1617:28vemvNice! Looking forward :)#2024-03-1620:58escherizeAmazing! How does this work?#2024-03-1705:52Lari SaukkonenWith malli I get subschemas with unique value paths, extract the right side name, in this case imaginary-subfield from the call, then find value paths containing it and return next values as a completion candidates.
It is a compliment custom source that I use with cider-nrepl-middleware to return completion candidates to cider and emacs. It should work also for anything using cider-nrepl-middleware if they use it for completions but im not sure if showing just these if they are found is as easy.
Programming with schemas has the example for value-paths and subschemas:
https://github.com/metosin/malli?tab=readme-ov-file#programming-with-schemas
Compliment has docs about custom-source:
https://github.com/alexander-yakushev/compliment/wiki/Custom-sources
Here cider-nrepl-middleware registers compliment sources, I had to change this a bit to add my custom source:
https://github.com/clojure-emacs/cider-nrepl/blob/master/src/cider/nrepl/middleware/complete.clj#2024-03-1709:32vemvFrom the other thread I was aware that you were hacking something with Compliment, but it makes me happy that it's for Malli specifically - I'm a fan!
Bridging Compliment contexts and Malli subschema walking would seem a great innovation.
It could mean that, for instance, besides from completion, I could "jump to definition" for :id , where :id is dependent on the Malli schema present in the current Compliment context.#2024-03-1709:34vemvI'm also interested in showing/walking Malli schemas interactively (in CIDER we have a "spec browser" with no Malli counterpart)#2024-03-1806:41ikitommilooks awesome!#2024-03-1812:53Lari SaukkonenThanks, I’m glad that it was something that might add value in the future for someone else also. 🙂
I plan to start using this daily and expanding the variations to destructuring etc. and see what kind of corner cases and things I will bump into.#2024-03-1806:43ikitommithere are many new PRs and Issues I haven’t had time to check. Have had busy few weeks, taking friday off to go through ’em and work on my wip-malli things#2024-03-1811:55MariusHi all! I’m pondering about how/whether to pass external parameters to a custom Fn schemas. Something like “current date” (which is not necessarily the real date). It seems I would have to add it to the data to be validated, although from a logical point it does not belong there. Any ideas?#2024-03-1813:39EdNot sure if malli supports this directly, but you could use dynamic scope to work around that? See https://clojure.org/reference/vars#2024-03-1813:50MariusThat would work, but unfortunately the environment is not the same for all validations#2024-03-1815:04Ednot sure what you mean by "environment", but dynamic scope should let you control it at the call-site like an extra parameter to your validation function.
(require '[malli.core :as m])
(def ^:dynamic *size-limit-factory* (fn [] 10))
(def too-big? #(< (*size-limit-factory*) %))
(def my-schema [:fn (fn [x] (not (too-big? x)))])
(m/validate my-schema 8) ;; => true
(m/validate my-schema 12) ;; => false
(binding [*size-limit-factory* (fn [] 100)]
[(m/validate my-schema 80)
(m/validate my-schema 120)]) ;; => [true false]
otherwise, you'll need to detect the env inside the validation function?#2024-03-1815:15MariusAh ok, I didn’t know that it works like that. My assumption was that the var will be a global var. Thanks for the explanation!#2024-03-1815:16MariusIf there was a button to upvote or give kudos, you’d earned it 🙂#2024-03-1815:20Ed😁 ... just be careful, dynamic scope is generally considered bad api design because it often breaks the principal of least surprise. It's a hidden argument to every function that uses it.#2024-03-1815:26Mariuswill do. thanks for the heads up#2024-03-1908:20roklenarcicDo yall use :sequential or do you use :* or seq? for your collection schema?#2024-03-1910:52Noah BogartSequential is for sequences of items, vectors or lists or lazy seqs. : is a meta schema, it modifies how the enclosing schema is interpreted. So [:cat :int [: :int]] means [:or [:cat :int] [:cat :int :int] [:cat :int :int :int] …]#2024-03-1910:53roklenarcicIs there a way I can alias :sequential into something shorter in my app then?#2024-03-1913:03Noah BogartThere's probably a way to do aliases, but I dn't remember at the moment#2024-03-1916:52ambrosebs@UEENNMX0T :* is equivalent to :alt not :or right?#2024-03-1916:53Noah BogartOh maybe, i forgot about :alt lol#2024-03-1916:53Noah BogartWhat’s that do again?#2024-03-1916:53ambrosebsI think it's a regex :or.#2024-03-1916:55ambrosebs:alt allows you to continue nesting in the same regex IIRC#2024-03-1916:56Noah BogartAh okay. I was trying to avoid saying regex cuz i didn’t wanna fall down an explanation hole#2024-03-1916:56Noah Bogart“What’s a regex schema?” Then i start sweating “ummm”#2024-03-1908:21roklenarcicWhat’s the downside of using :* instead of :sequential?#2024-03-1917:02ambrosebsyou need to be careful where you nest :* because it means something different inside sequence regexes.#2024-03-2016:00escherizeperformance is generally worse with :*, I think. If you aren’t doing regex-schemas I would stick to using :sequential.#2024-03-2008:35mkvlris there a way to specify what keys a map has to contain, without saying what that key is?
(malli.core/validate [:map [:my/uuid uuid?]] {:my/uuid (random-uuid)}) ;; works
(malli.core/validate [:map {:registry {:my/uuid uuid?}} :my/uuid] {:my/uuid (random-uuid)}) ;; also works
(malli.core/validate [:map :my/uuid] {:my/uuid (random-uuid)}) ;; throws :malli.core/invalid-schema#2024-03-2015:59escherizeyou mean like [:map [:my/uuid :any]] ?#2024-03-2015:59escherize[:map :my/uuid] is not a valid schema, there#2024-03-2016:00mkvlrthat’s sort of my question#2024-03-2016:00mkvlrcan I not specify that the key is needed, without saying what is in the key?#2024-03-2016:01escherizeI think this answers it: https://clojurians.slack.com/archives/CLDK6MFMK/p1710950351557149?thread_ts=1710923722.360299&cid=CLDK6MFMK#2024-03-2016:01escherizeyou can, and that is how#2024-03-2016:01mkvlrthen I do say that the value is any?#2024-03-2016:02escherizeI must not understand your question#2024-03-2016:07mkvlras I understand it, clojure.spec made the intentional choice to separate what keys are in a map, from what the contents of these keys are, see https://clojure.org/about/spec#_map_specs_should_be_of_keysets_only and https://clojuredocs.org/clojure.spec.alpha/keys. I was wondering if there’s a similar construct in malli.#2024-03-2016:08mkvlrI can’t find a way to specify a key being required without a spec.#2024-03-2016:09escherizewhat use case does putting :any not cover for that? an example will help#2024-03-2016:10mkvlrit doesn’t reflect what I actually want to say, which is I may or may not specify this spec elsewhere, but it probably won’t be any?#2024-03-2016:13mkvlr(I haven’t tried yet how [:map [:my/uuid any?]] would interact with the global registry)#2024-03-2018:16ambrosebsThis looks relevant https://github.com/metosin/malli?tab=readme-ov-file#qualified-keys-in-a-map#2024-03-2018:22ambrosebsI don't think malli checks namespaced keys that aren't declared in the :map though.#2024-03-2018:23ambrosebsi.e., I don't think it has this feature of keys:
In addition, the values of *all* namespace-qualified keys will be validated
(and possibly destructured) by any registered specs.
#2024-03-2018:25mkvlryes, the qualified keys in a map example with the local registry is my second example but it fails without a registry argument#2024-03-2018:26ambrosebsYes sorry didn't fully understand your example at first.#2024-03-2018:37ambrosebsLooking at the implementation of :map I don't think this is supported by :map. Could do this:
user=> (m/validate [:and [:map] [:fn `#(contains? % :my/key)]] {})
false
user=> (m/validate [:and [:map] [:fn `#(contains? % :my/key)]] {:my/key true})
true
#2024-03-2018:37ambrosebsThe generator would need tweaking though.#2024-03-2020:21mkvlr@U055XFK8V is :map a generator? Do you think supporting (malli.core/validate [:map :my/uuid] {:my/uuid (random-uuid)}) could be a sensible addition to the syntax?#2024-03-2020:29ambrosebsI mean that trying to generate values will fail:
user=> (mg/generate [:and [:map] [:fn `#(contains? % :my/key)]])
Execution error (ExceptionInfo) at clojure.test.check.generators/fn (generators.cljc:435).
Couldn't satisfy such-that predicate after 100 tries.
Will need a property like :gen/fmap to fix it:
user=> (mg/generate [:and [:map {:gen/fmap `#(assoc % :my/key :whatever)} ] [:fn `#(contains? % :my/key)]])
#:my{:key :whatever}
Could probably make a case for at least supporting a property for the behavior you want, like say [:map {:unregistered-qualified-kw true} ...] .#2024-03-2020:35mkvlrah right, thanks for the explanation!#2024-03-2009:03mkvlralso noticed when running the example from https://github.com/metosin/malli/blob/master/docs/function-schemas.md#development-instrumentation that I needed to add a call to (mi/instrument!) for the example to throw. Is that expected? This was with malli 0.11, trying 0.14… Same there.#2024-03-2108:16Tommi MartinWould someone have examples of dispatch usage for malli that i could use to try to understand all the possibilities with it? I'm trying to improve default -> or -> [map-of|map-of|string] declaration to get better errors for myself. But I'm struggling with the nested structure and the lack of a single field that actually differentiates the maps from each other.#2024-03-2108:29Tommi MartinOver all i'm not even sure I should be using the default system to achieve the effect but i'm not sure how else to do it.
My schema is a map. This map then has several fields with string values. Followed by two nested maps. Indexed by the names of those nested maps. These nested maps then have one more level of nesting that contains the data i'm reading. so it the structure looks like this
{:name "first level"
:field1 "value"
:field2 "value"
:random-name {:random-name2 {:name "submap"}}
:random-name3 {:random-name4 {:name "the other submap"}}}
At the moment i'm achieving malli validation for that with something like this:
[::m/default
[:or
[:map-of {:gen/min 1 :gen/max 1} :keyword
[:map-of {:gen/min 1 :gen/max 3} :keyword
[:map
[:name "submap-1"]]]]
[:map-of {:gen/min 1 :gen/max 1} :keyword
[:map-of {:gen/min 1 :gen/max 3} :keyword
[:map
[:name "submap-2"]]]]
[:string]]]
this works for validation. But it doesn't work for data generation. When both of the maps should be present. I would like to know if there is a better way of achieving this. The example is a bit truncated and have the other different fields between the two maps to keep the examples as short as possible. But in reality the submap-1 and 2 have a different set of fields they validate slightly differently#2024-03-2109:54jussiWe use dispatch like this. We have defined, mostly for clarity and re-usability, our schema in multiple smaller entities that are used to compose the final schema.
(def leg-event
(mu/closed-schema
(m/schema
[:multi {:dispatch :vehicle
:description "A discrete part of a trip, using one vehicle to go directly from an origin to a destination (i.e. without connecting flights). Allowed values are; flight, car, ferry and train"}
["flight" (m/deref-all flight {:registry registry})]
["ferry" (m/deref-all ferry {:registry registry})]
["train" (m/deref-all train {:registry registry})]
["car" (m/deref-all car {:registry registry})]]
{:registry registry})))
You can then dispatch the schema like
(m/validate leg-event
{:vehicle "train"
:stops ["Helsinki" "Oulu" "Kittilä"]
:distance_traveled_in_km 300})))
Not sure if this is what you are looking for but this allows certain schema validation to be triggered based on value in data.#2024-03-2110:40Tommi MartinIt's close to what I'm looking for. But unlike in your example where you can dispatch :vehicle I don't similar field that is the only source of truth with a known value I could dispatch on. Basically my dispatch would need a function to determine what to dispatch with the data I have. Kind of like
(if (map? value)
(if (source-fields-exist? value)
(dispatch :data-source)
(dispatch :data-service)
(dispatch :string)))
If I kept using the m/default I think I would have to have a logic similar to the dirty if statement above to dispatch the correct value. But I'm not sure I can do that with Dispatch. or if I even should attempt such a hack.#2024-03-2110:42Tommi MartinI could compute a dispatchable field before i feed the current code with the data but that feels a bit wonky in the architecture as the validation I am doing is an ingest validation before transforming the data to a different format. so manipulating the ingest data before it's transformed doesn't fit cleanly in the app i've built. But it might be the only option here#2024-03-2110:55jussiMaybe you can use a function, it says https://github.com/metosin/malli?tab=readme-ov-file#multi-schemas that any function can be used for dispatch
Any function can be used for :dispatch:
(m/validate
[:multi {:dispatch first}
[:sized [:tuple keyword? [:map [:size int?]]]]
[:human [:tuple keyword? [:map [:name string?] [:address [:map [:country keyword?]]]]]]]
[:human {:name "seppo", :address {:country :sweden}}])
; true#2024-03-2111:01Tommi MartinI'll see if i can use a custom function over inbuilt ones, thank you kindly for your input.#2024-03-2313:11ikitommiMerged all done stuff, so welcome`[metosin/malli "0.15.0"]`! Changes below:
> • :=> takes optional 3rd child, the guard schema validating vector of arguments and return value [args ret]. See https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-guards for more details. Fixes https://github.com/metosin/malli/issues/764 and https://github.com/metosin/malli/issues/764.
> ;; function of arg:int -> ret:int, where arg < ret
> [:=>
> [:cat :int]
> :int
> [:fn (fn [[[arg] ret]] (< arg ret))]]
> • BREAKING: malli.generator/function-checker returns explanations under new keys:
> ◦ ::mg/explain-input -> ::m/explain-input
> ◦ ::mg/explain-output -> ::m/explain-output
> ◦ new ::m/explain-guard to return guard explanation, if any
> • m/explain for :=> returns also errors for args, return and guard if they exist
> • FIX m/deref-recursive doesn’t play nice with :merge schema https://github.com/metosin/malli/issues/997 via https://github.com/metosin/malli/pull/999
> • FIX nested :repeat sequence schema’s doesn’t seem to work https://github.com/metosin/malli/issues/761 via https://github.com/metosin/malli/pull/1024
> • FIX Invalid Swagger JSON with [:or :nil] alternatives https://github.com/metosin/malli/issues/1006 via https://github.com/metosin/malli/pull/1023
> • FIX (explain :tuple []) https://github.com/metosin/malli/pull/1022
> • Enforce entry specs in open map destructurings https://github.com/metosin/malli/pull/1021
> • FIX goog/mixin was deprecated and is now removed https://github.com/metosin/malli/pull/1016
> • Updated dependencies:
> borkdude/edamame 1.3.23 -> 1.4.25
#2024-03-2414:353starblazeWhat would be the preferable way to document schemas? I tried searching in the Malli's readme, Malli's issues and this Slack instance but couldn't find anything. Should I just attach :doc metadata to the corresponding item?
Here is a practical example. I have this type in a registry (it's in a registry because it's a recursive type) and I would like to attach documentation "Version when this function was introduced" to ::arg-docs and "parameter description" to ::arg-docs ::name.
{::lib-fn
[:map
[::return-type [:ref ::type]]
[::args [:sequential [:ref ::type]]]
[::since :string]
[::arg-docs [:sequential
[:map
[::name ::identifier]
[::doc :string]]]]]}#2024-03-2417:20ikitommiI would use :title and :description properties. They also get propagated to https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc#L29.#2024-03-2417:20ikitommi> The title and description keywords must be strings. A “title” will preferably be short, whereas a “description” will provide a more lengthy explanation about the purpose of the data described by the schema.#2024-03-2417:20ikitommihttps://json-schema.org/understanding-json-schema/reference/annotations#2024-03-2419:523starblazeSomething like [::since {:description "Version when this function was introduced"} :string]?#2024-03-2507:35Adham OmranI'd like help understanding this :errors section of m/explain return.
({:path [0 :value 0], :in [5 :value], :schema :double, :value 89.86}
{:path [0 :value 1], :in [5 :value], :schema :nil, :value 89.86}
{:path [],
:in [5],
:schema
[:* [:map [:y :int] [:yName :string] [:value [:or :double :nil]] [:x :int] [:responders :int]]],
:value
{:y 8,
:yName "Master's graduate or equivalent",
:value 89.86,
:name "Communication and Collaboration",
:x 1,
:responders 5},
:type :malli.core/input-remaining})
Using me/humanize returns ["invalid function"] so that's no help#2024-03-2508:03Adham OmranThe function form works i.e. [:value [:or float? nil?]]#2024-03-2519:02escherizeI think you want to use :maybe instead of or nil? I’m not at my machine but I don’t think :nil is a valid schema. You can check with (m/schema :nil). #2024-03-2608:50Adham Omran(m/schema :nil) returns :nil and is mentioned https://cljdoc.org/d/metosin/malli/0.15.0/doc/readme#mallicoretype-schemas so it seems valid unless I am missing something#2024-03-2608:073starblazeI found a closed issue https://github.com/metosin/malli/issues/872 that says that Malli does not report issues about invalid schemas. It seems like this problem belongs to https://github.com/metosin/malli/issues/18 . If this is not being fixed right now, I could try to fix the issue myself by making a schema that validates schemas. If we don't count malli's readme, is there any good resource where I can get a precise description about all schemas and what they expect?#2024-03-2615:06respatializedyou don’t necessarily need to do that- if you call malli.core/schema on an invalid schema it will throw an exception that has some context#2024-03-2615:333starblazewell, I have my schemas defined in a registry map and when I check out with some simple examples, everything seems to be working but once I try to instrument all the functions (that I typed via metadata), I get malli's child errors that don't tell much. It just says something about :enum but I only use it once and it has options, so I am not sure why it would complain.
:data {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :enum, :properties nil, :children nil, :min 1, :max nil}}#2024-03-2617:493starblazeafter some painful debugging I realized what went wrong. I put :enum in registry because I thought I will need a bunch of types and I should probably put them in a registry. It seems that anything that is redefined in registry never get any parameters, so schemas like :enum , :or, :not that expect at least one argument will fail. I am not sure that's the expected behavior#2024-03-2617:493starblaze(I defined that in a local registry if that changes things)#2024-03-2617:563starblazeHere's a minimal example
(m/schema [:enum "foo" "bar"]) ;; ok
(m/schema [:enum "foo"] {:registry {:enum (:enum (m/default-schemas))}}) ;; ok
(m/schema [:enum {:registry {:enum (:enum (m/default-schemas))}} "foo" "bar"]) ;; not ok
{:type :malli.core/child-error,
:message :malli.core/child-error,
:data {:type :enum, :properties nil, :children nil, :min 1, :max nil}}#2024-03-2609:52pezIs there a way to express at-least-on-of these keys for a map? I have a padding:
[:map
[:top {:optional true} number?]
[:bottom {:optional true} number?]
[:left {:optional true} number?]
[:right {:optional true} number?]]
Where it’s fine if either of the sides is mentioned, but an empty map is probably a mistake.#2024-03-2610:56Stig BrautasetWith a bit of duplication:
[:or
[:map
[:top number?]
[:bottom {:optional true} number?]
[:left {:optional true} number?]
[:right {:optional true} number?]]
[:map
[:top {:optional true} number?]
[:bottom number?]
[:left {:optional true} number?]
[:right {:optional true} number?]]
[:map
[:top {:optional true} number?]
[:bottom {:optional true} number?]
[:left number?]
[:right {:optional true} number?]]
[:map
[:top {:optional true} number?]
[:bottom {:optional true} number?]
[:left {:optional true} number?]
[:right number?]]]
see the Malli http://playground.io#2024-03-2610:56Stig BrautasetI can't think of another way without using :fn schemas.#2024-03-2611:03pezThanks, @U03N9E40Q2F 🙏#2024-03-2611:443starblazeHow about [:and :YOUR_SCHEMA [:not [:map {:closed true}]]?#2024-03-2611:57Stig BrautasetOh, that's nice. How does that work? 😄 https://malli.io/?value=%7B%7D&schema=%5B%3Aand%0A%20%5B%3Amap%0A%20%20%5B%3Atop%20%7B%3Aoptional%20true%7D%20number%3F%5D%0A%20%20%5B%3Abottom%20%7B%3Aoptional%20true%7D%20number%3F%5D%0A%20%20%5B%3Aleft%20%7B%3Aoptional%20true%7D%20number%3F%5D%0A%20%20%5B%3Aright%20%7B%3Aoptional%20true%7D%20number%3F%5D%5D%0A%20%5B%3Anot%20%5B%3Amap%20%7B%3Aclosed%20true%7D%5D%5D%5D#2024-03-2611:57Stig Brautaset(I can see that it appears to, but I don't understand how.)#2024-03-2611:593starblazeMap has an attribute :closed and when it's set to true it rejects all maps that have extra keys specified. Since we didn't specify any key, it means that it will reject everything except an empty map and that's why we add :not#2024-03-2612:033starblazeYou could also reject empty maps with [:not [:map-of [:not :any] :any]] because map-of asks all keys to satisfy something impossible and that will happen on empty maps because there is nothing to check, so that means that all keys satisfy the impossible. Anyhow this is a really confusing example but it is kind of fun to look at.#2024-03-2612:25pezI think the first one is pretty clear. At least after you explained it. 😃#2024-03-2612:26pezIs your http://malli.io link the link you tried to give me first, @U03N9E40Q2F? That one didn’t work, but the http://malli.io one sure looks like a playground.#2024-03-2615:26Stig Brautaset@U0ETXRFEW ah, I seem to have messed up the paste somehow. The first link I gave was to http://malli.io with the example I gave. Which works, but I think @U02S4QZAH61’s example is much neater.#2024-03-2706:52ambrosebsI took a stab at adding keyset constraints to malli https://github.com/metosin/malli/pull/1025
Here's how you'd write padding with that:
(def Padding
[:map
{:keys [[:or :top :bottom :left :right]]}
[:top {:optional true} number?]
[:bottom {:optional true} number?]
[:left {:optional true} number?]
[:right {:optional true} number?]])
(m/validate Padding {:left 1 :right 10 :up 25 :down 50})
;=> true
(me/humanize
(m/explain Padding {}))
; => ["should provide at least one key: :top :bottom :left :right"]
(mg/sample Padding {:size 5})
; => ({:top -0.5} {:top -1} {:left 0} {:left -1} {:left 1.0})
#2024-03-2707:12pezVery cool @U055XFK8V! As a malli noob I wonder if this points at a way to swap the semantics to mean that entries are optional by default. So something like
[:map
{:req [[:or :id :name]]}
[:id :int]
[:name :string]
[:speed number?]
[:color :style/color]]
Would mean:
> If you give a map of valid either :id or :name, I’m good, as long as any :speed, or :color also provided checks out.
Obviously, I don’t know what I’m doing, just a thought that popped to mind.#2024-03-2707:163starblaze@U055XFK8V Isn't this a confusing behavior since I am used to fact that normally :or's children are schemas. If we only want to constrain the keys maybe it would make more sense to conform to existing behavior. Perhaps something like [:cat [:* :any] [:enum :top :left :bottom :right] [:* :any]]. I have heard that seqexps are relatively expensive, so perhaps it would be a better idea to include something like [:exists SCHEMA] which is true if the collection has an element that conforms SCHEMA.#2024-03-2707:183starblaze@U0ETXRFEW oh yeah, my solution might not be exactly what you need because it can accept {:botom 3} (with one t) or any other map that has something in it. My solution would work fine if your entire schema would be closed.#2024-03-2707:21pezFor the record, upon closer inspection my particular use case doesn’t need this. It turns out it’s fine with all dimensions lacking. Still needed to learn about this, so am very happy for all the info in this thread!#2024-03-2709:44Ben SlessI still believe what we need is symbolic schema manipulation / simplification#2024-03-2709:47pez@UK0810AQ2 is there somewhere I can read about symbolic schema manipulation / simplification?#2024-03-2709:51Ben SlessBoolean / logical simplification, SAT solvers, type lattice#2024-03-2709:52Ben SlessThink
or (and a b) (and a c) can be (and a (or b c))#2024-03-2709:53Ben SlessWhat are map entries if not a big and#2024-03-2709:543starblazeDoesn't malli do some optimizations under the hood?#2024-03-2709:59pezWith such manipulation in place, how would the OP use case be solved? And if it also solves my wish for swapping the required semantics, that would also be nice to see an example of. 😃#2024-03-2713:30Noah Bogart@U055XFK8V that looks very nice, we'd use that at my job. the nested vectors on :keys is a little strange, what's the reason for it?#2024-03-2714:32ambrosebs@UEENNMX0T thanks! it's the syntax of s/keys but datafied:
(s/keys :req [::x ::y (or ::secret (and ::user ::pwd))] :opt [::z])
=>
[:map {:keys [[:or ::secret [:and ::user ::pwd]]]}
::x ::y [::pwd {:optional true}]
[::secret {:optional true}]
[::user {:optional true}]
[::pwd {:optional true}]
#2024-03-2714:33Noah Bogartoh that makes sense#2024-03-2714:37ambrosebs@U02S4QZAH61 it's a good point that :or etc is overloaded. spec is the inspiration ^. Sequence expressions are for sequential things, and keysets are not sequential, so I'm not sure there's existing work for this.#2024-03-2714:39ambrosebs@U0ETXRFEW yeah optional by default if you give keyset constraints would help with succinctness. I'm not sure it can be bolted on entirely cleanly, maybe we need a new schema.#2024-03-2714:413starblaze@U055XFK8V oh yeah, I am just so used to the fact that maps (and its keys or values) can be converted to sequences and can be fed to various Clojure functions.#2024-03-2714:48ambrosebs@U02S4QZAH61 I recently made a schema that checks a map based on a sequence expression. It's very odd, it doesn't quite make sense, but could almost be the seeds of something useful. Here's the equivalent to [:map-of :int :int].
[:into-map
[:* [:tuple :int :int]]]
#2024-03-2714:49Noah Bogarti wonder if borrowing the :keys syntax from spec duplicates too much from existing malli behavior, instead of adding a new key to the properties map on :map: [:map {:or [:top :bottom :left :right]} [:top ...] [:bottom ...] ...]#2024-03-2714:51ambrosebs@UEENNMX0T that's cool! :keys would become :and, and everything is more succinct.#2024-03-2714:533starblaze@U055XFK8V if we had some set predicates built-in, we could treat keys as a set and use those set predicates on them. Something like includes? (that could also benefit sequences).#2024-03-2714:543starblazeI guess all set predicates could be used on sequences, so maybe there is not a big point differentiating between them.#2024-03-2714:543starblazeoh yeah, it's called contains?, pardon my Clojure#2024-03-2714:57ambrosebs@U02S4QZAH61 I think there might be something to the DSL I created for keysets that could be generalized. It's a constraint language about set inclusion, the atomic constraint is #(contains? % k).#2024-03-2714:57ambrosebs[:contains [:or :secret [:and :user :pass]]]#2024-03-2715:033starblazecontains? is for literals. I don't know what would be the Clojure-y way but there should be some collection predicate that says "if there is a member of x that satisfies f, return true" normally I would call that any? but it means a different thing in Clojure.#2024-03-2715:03ambrosebsYeah I guess that might be another atomic constraint.#2024-03-2715:05ambrosebsIt might be a similar situation to why ::m/default was added to :map.#2024-03-2715:05ambrosebsYou attach an extra schema for everything not specified.#2024-03-2715:063starblazeIsn't it the same thing as [:and [:map ...] [:map-of X Y]?#2024-03-2715:07ambrosebsYeah. I'm sort of seeing a correspondence between :contains <=> :map and :set <=> :map-of . That's why I thought of whether a ::m/default could be relevant.#2024-03-2715:093starblazethe one problem with :map-of is that it does not have a context about the entire key set or value set (values are not guaranteed to be a set but yeah)#2024-03-2715:103starblazeIf you haven't read it yet, you might want to check out my idea about schema-validation. It could also open doors to schemas that accept arguments (that could mean generics)#2024-03-2715:113starblaze(I recently started learning Haskell, types have bad influence on me)#2024-03-2715:11ambrosebsNice, I'm in the middle of implementing type functions for malli actually.#2024-03-2715:15ambrosebsfiguring out the locally nameless representation right now so I can implement capture-avoiding substitution. but here's what it might look like:
;; spec for clojure.core/map transducer
[:schema {:registry {"Reducer" (m/tfn [a b] [:=> [:cat b a] b])
"Transducer" (m/tfn [in out] (m/all [r] [:=> [:cat [:ref "Reducer" out r]] [:ref "Reducer" in r]]))}
(m/all [a c] [:=> [:cat [:=> a c]] [:ref "Transducer" a c]])]#2024-03-2715:183starblaze:ref can take multiple arguments?#2024-03-2715:19ambrosebsThankfully not! so I can extend it 🙂#2024-03-2715:203starblazeoh, I think :ref always takes just 1 one argument, at least that's what I remember and what I see in the readme. What are those symbols after the first child? Are they typenames?#2024-03-2715:21ambrosebs@U02S4QZAH61 what I just wrote is part of my WIP branch. Sorry I have a habit of dropping wishful thinking and causing confusion.#2024-03-2715:213starblazeoh yeah, I figured out, it's a WIP schema, I am just not sure what these would represent#2024-03-2715:23ambrosebsI have a simple macro (m/tfn [x y]...) that just does (let [x 'x y 'x] ...). Then you get a schema like '[:tfn [x y] [:=> ... x ... y]].#2024-03-2715:23ambrosebsAnd the symbols are polymorphic type variables scoped over the body.#2024-03-2715:24ambrosebsyou can then instantiate them with :ref (or maybe another operation like :inst that I haven't decided on).#2024-03-2715:253starblazeah okay, understood#2024-03-2715:26ambrosebsI did an introductory talk on the basic idea here FWIW https://www.youtube.com/watch?v=QE3PZLlefUE#2024-03-2715:26ambrosebsand the WIP branch is here https://github.com/frenchy64/malli/pull/2/files#2024-04-0313:36Daniel Galvini do this type of thing with a fn
(defn at-least-one-key?
[keys]
[:fn {:error/message (str "At least one of " keys " should be supplied")}
(fn [data]
(seq (select-keys data keys)))])
then i have my schema defined like
(def fcm-notification-body
[:and
[:map
[:title {:optional true} non-empty-string]
[:body {:optional true} non-empty-string]
[:image {:optional true} url?]
[:data {:optional true} [:map-of :keyword :string]]]
(at-least-one-key? [:title :body :image :data])])
Apologies if this was solved already but felt it was a nice example#2024-04-0313:43pezThanks, @U069RSXM1CP! I’ll remember this recipe. Though, I do like data better than functions, so would be nice with a declarative solution.#2024-04-0313:47Daniel Galvin👍 yeah i was trying to do it declaritively at first and had things like chunky :or with repeated stuff.
might be able to be expressed by saying its a closed map and not empty but havnt fiddled much, once i got used to using the :fn stuff ive done it alot.
You can describe custom generation logic if that is the concern 😄#2024-04-0313:52pezIt’s more that my default mode is something like#2024-04-0314:10Daniel Galvini thought i was really clever for a second with this type of idea
[:and
[:map {:closed true}
[:keyone {:optional true} :int]
[:keytwo {:optional true} :int]]
[:not [:map {:closed true}]]]
but was mentions above already, works tho (ie, 1st schema is the real one, 2nd is an empty map, basically saying, To be valid, you must NOT be empty, and you MUST provide a key that is valid
it doesnt really make it so you can have various conditionals tho, like IF keyone is over 100, then keytwo must be below 50.#2024-03-2618:213starblazeHow could I find the offending function name when instrumenting defns?
For example I got a :malli.core/invalid-output exception with the offending value and function signature but I do not get informed about the function itself. I don't have many functions, so I can deduce the offender by its signature but it seems like a clumsy way of handling defn signature violations.#2024-03-2623:30tlonisthow do I attach custom transformers to malli schema for coercion?
My end goal is to define transformers for coercion at the schema level, then have the reitit.malli.coercion take care of the coercing whenever requests are received.#2024-03-2702:04tlonist(m/coerce
[:map
[:x [:bigdec? {:decode/string->bigdec {:enter #(bigdec %)}}]]
[:y [:time/instant {:decode/string->instant {:enter str->inst}}]]]
{:x "123"
:y "2023-01-01T00:00:00Z"}
(mt/transformer
{:name :math}
{:name :string->instant}))
this works (with str->inst being a function defined in ns), but I need to know how this can be applied when combined with reitit's malli coercion.#2024-03-2706:52ambrosebsI took a stab at adding keyset constraints to malli https://github.com/metosin/malli/pull/1025
Here's how you'd write padding with that:
(def Padding
[:map
{:keys [[:or :top :bottom :left :right]]}
[:top {:optional true} number?]
[:bottom {:optional true} number?]
[:left {:optional true} number?]
[:right {:optional true} number?]])
(m/validate Padding {:left 1 :right 10 :up 25 :down 50})
;=> true
(me/humanize
(m/explain Padding {}))
; => ["should provide at least one key: :top :bottom :left :right"]
(mg/sample Padding {:size 5})
; => ({:top -0.5} {:top -1} {:left 0} {:left -1} {:left 1.0})
#2024-03-2708:063starblazeI'd appreciate if people could give me their opinion about this. Related to https://clojurians.slack.com/archives/CLDK6MFMK/p1711440465695169 where I asked about schema validation.
When I was struggling to find out an issue about why my schema was failing, I was thinking about a solution that clearly reports what is wrong. I don't know much about Malli's architecture so I might be doing something that already exists.
My proposed architecture is following. We split schema into three parts: schema-id, attributes and children. We check if the schema-id is in the registry and if it is not, we report that schema-id is not in registry. Othwerwise we check if we can resolve it. There are two cases: it's a built-in schema (e.g. :map, :string, :or) or it's a user-defined schema in a registry. If it's user defined, we reduce to built-in schemas. Then we check built-in schemas.
How? We have malli schemas for each schema. Each schema gets a tuple of [attrs children] and we use malli itself to validate a schema. Here are two schemas for :map and :string that I made for testing purposes.
(def schema-resolution
"Key is a schema identifier, value is the schema that takes [attrs children].
When children/attrs are not explicitly defined they are nil, when they are defined
but are empty they are []/{} respectively."
{:map [:tuple [:maybe :map] [:maybe [:sequential [:tuple :any ::schema]]]]
:string [:tuple [:maybe :map] [:maybe [:sequential {:max 0} :any]]]})
I am not handling attrs right now, so we can forget about them for a moment. The most interesting part is ::schema which recursively allows constraining an item to a schema.
I have a small prototype that can correctly check schemas that only use :map and :string with arbitrary levels of nesting, though I am struggling a bit with the error reporting because under the hood I inject {::schema [:fn SOME-FN]} which splits apart schemas and reports a different location if error is encountered.
With this we could:
- get explanation about why schema is not accepted
- make attribute checking and report extra keys (e.g. if someone passed :max attribute to :map, a validator could warn the developer that it did not expect that key)
- implement some style checks like
- useless brackets (prefer :smth instead of [:smth])
- (prefer [:= CONST] instead of [:enum CONST])
- impossible conditions [:and :string :int]
- (not directly related to this but) detect illegal recursion
- a schema can reach itself without going through :ref (e.g. {::foo [:tuple :string [:map [:k [:maybe ::foo]]]]})
- a recursive schema with :ref that cannot terminate (e.g. {::foo [:tuple :string [:ref ::foo]]})#2024-03-2715:30ambrosebsHow does your work relate to this comment? https://github.com/metosin/malli/issues/872#issuecomment-1470751033#2024-03-2715:383starblazeI glossed over that comment because it was said that there are no actual schemas defined. It looks like there is support to add metaschemas but no schema implements that. If it's simple to obtain properties and children and I can use the same vector syntax I have been using, I could definitely make a PR and fill those gaps. If I implemented metaschemas for all current schemas, would that also give automatic schema error reports?#2024-03-2715:39ambrosebsI'm not sure, does anything call it?#2024-03-2715:413starblazeHm, not sure. We could just call m/explain on a schema with meta-schema and everything should go fine.#2024-03-2715:45ambrosebsZooming out, my hunch is that each bad error message is the fault of the particular IntoSchema implementation that throws it. And we need a default case for parsing schemas that don't exist. And some of the things you listed are semantic issues not syntactic, they would be implemented using the already-parsed schema. e.g., you need access to the full registry to detect an infinitely expanding schema.
I agree that error messages are often bad but I think we're missing a step where we compile a list of all the bad error messages and see what the best approach is to fixing them.#2024-03-2715:483starblazeI will probably study malli internals at some point to get a better idea how to apply this. When referencing something that does not exist, malli just throws invalid schema.#2024-03-2715:483starblazeI also had a bug where I pulled in a bunch of built-in schemas and it started to give wrong errors (like my schema calling :enum with no children)#2024-03-2715:49ambrosebsIt would be really valuable if you could compile these bad error messages.#2024-03-2715:503starblazeHere's a minimal :enum bug example https://clojurians.slack.com/archives/CLDK6MFMK/p1711475816145789?thread_ts=1711440465.695169&cid=CLDK6MFMK, it probably should be turned into an issue on github, I'm not sure.#2024-03-2715:51ambrosebsAh, :enum doesn't support properties?#2024-03-2715:523starblazeNo, the error implies I used :enum and not [:enum "foo" "bar"]. You can see it says that there are no children and that I have to provide at least one#2024-03-2715:523starblazeI mean enum probably supports properties but it's not about properties#2024-03-2715:543starblazeMy guess is that malli knows that :enum is a local registry key and assumes that it can't have any properties or children (because custom schemas are just aliases and at this point they don't support any custom behavior)#2024-03-2715:55ambrosebsThis is thrown by m/-check-children! which is often the only error checking for schema children. I can't figure out what it's complaining about here.#2024-03-2715:563starblazeas I said in my previous comment, malli probably discards children and properties of local registry references#2024-03-2715:573starblazeand then that stripped information is passed to a validator which starts to complain#2024-03-2715:59ambrosebsI don't think you're allowed to put an IntoSchema into a property registry, only options. You can't deserialize it.#2024-03-2716:00ambrosebsOh dear, because an IntoSchema is a map, malli tries to parse it as {:type :enum} .#2024-03-2716:02ambrosebsEh no that seems wrong, there's an IntoSchema case.#2024-03-2716:05ambrosebsWell this is the minimal case:
user=> (m/schema (:enum (m/default-schemas)))
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
:malli.core/child-error
I think IntoSchema's in options might be special, perhaps they're not coerced into schemas. I know property schemas get coerced.#2024-03-2716:06ambrosebsSay that's true, perhaps we could add a case in -property-registry to blow up if you provide an IntoSchema? That's the kind of idea I was suggesting instead of making metaschemas.#2024-03-2716:073starblazeThe schema looks like it is working correctly, I think it just incorrectly interpreted my [:enum "foo" "bar"].#2024-03-2716:08ambrosebsif you look at the stacktrace you'll see that it correctly interpreted your schema, and is failing trying to coerce your registry to schemas.#2024-03-2716:083starblazeYou can see it says that an enum must have at least one child which is correct, the problem is that my enum has 2 children.#2024-03-2716:083starblazeyeah, I think Malli might assume that local registry references can't have children and options which is kinda logical because we don't have generics and that kind of stuff#2024-03-2716:09ambrosebsIt's not complaining about the outer schema.
user=> (m/schema [:= {:registry {:enum (:enum (m/default-schemas))}} "foo"])
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
:malli.core/child-error #2024-03-2716:093starblazeoh#2024-03-2716:11ambrosebsIf you want to try and fix this I'd suggest guarding the call to m/schema in m/-property-registry with a check for into-schema? so you can fail earlier. Worth trying at least.#2024-03-2716:123starblazealright, shall I raise an issue on Github?#2024-03-2716:123starblazenot sure when the fix will come, I need to check out Mallis internals a bit to get a better feel how Malli works under the hood#2024-03-2717:53ambrosebsI had a shot at a fix, could you try it out? https://github.com/metosin/malli/pull/1026#2024-03-2718:143starblazeYou want me to try the fix with extra examples?#2024-03-2718:55ambrosebsOr at least just a sanity check that I fixed the issue you found.#2024-03-2719:123starblazeYup, looks good at my side, I tested with these two calls:
(m/schema [:enum {:registry {:enum (:enum (m/default-schemas))}} "foo" "bar"])
(m/schema [:or {:registry (m/default-schemas)}
[:enum "foo" "bar"]
[:= "baz"]])#2024-03-2712:36andersmurphy(= (m/schema [:cat [:fn #{:div :span}] any? [:or vector? string?]])
(m/schema [:cat [:fn #{:div :span}] any? [:or vector? string?]]))
;; => false
So it looks like validator caching is based on object equality not value equality. Is this intentional?
(m/validator [:cat [:fn #{:div :span}] any? [:or vector? string?]])
@(m/-cache (m/schema [:cat [:fn #{:div :span}] any? [:or vector? string?]]))
;; => {}
(def foo
(m/schema
[:cat [:fn #{:div :span}] any? [:or vector? string?]]))
(m/validator foo)
@(m/-cache foo)
;; => {:validator #object[malli.impl.regex$validator...]}
(defn foo [tags]
[:cat [:fn tags] any? [:or vector? string?]])
(m/validator (foo #{:div})))
@(m/-cache (m/schema (foo #{:div})))
;; => {}
I’m effectively wanting to cache parameterised schemas.
So what I’d expect is the first time (m/validator (foo #{:div})) is called the validator is created and cached. Same if it’s called with a different value. eg (m/validator (foo #{:span})). Subsequent calls with those parameters would get the cached validators. Right now it looks like each call, even with the same parameter would create a separate entry in the cache rather than one for each unique value.
Am I missing anything? Is there a reason for this?#2024-03-2807:14ambrosebs@ikitommi could we move the :=> guard to properties (before 0.15.0) and consider more flexible argument syntax (later)? I think adding a guard as the final argument locks in the restriction that the args must be a :cat. These would be nice: [:=> ret], [:=> arg1 ret], [:=> arg1 arg2 arg3 ret] . But the guard would then be indistinguishable from the ret if it's part of the children.
There's a pointer here that seems to suggest it should be in the properties too https://github.com/metosin/malli/issues/608 (this should be referenced in the changelog, but there's a typo at the top of 0.15.0 that references the same issue twice).#2024-03-2814:11ikitommiThis was another option originally, but thought that having a new :-> that doesn’t support guards but would treat the last arg as return, wrap others in :cat would be cleaner for this. e.g. the following pairs would mean the same:
[:-> :int :int]
[:=> [:cat :int] :int]
:->
[:=> :cat :any]
[:-> :int]
[:=> :cat :int]
[:-> :int [:* :int] :int]
[:=> [:cat :int [:* :int]] :int]
#2024-03-2814:11ikitommihappy to hear your opinions on this.#2024-03-2814:13ikitommithat kind of extra schema would not be many lines of code.#2024-03-2815:06ambrosebs:-> sounds perfect.#2024-03-2815:09ambrosebsCould add support for a guard in properties too for feature parity.#2024-03-2815:10ambrosebs(I'm guessing :-> would just be sugar for :=> that just has a cleaner m/form )#2024-03-2815:28ambrosebsIt might be a good opportunity to consider non-cat args like [:=> [:* :int] :int] or [:=> [:alt :cat [:cat :foo :bar]] :any].#2024-03-2815:32ambrosebsIs there much code depending on :=> input being :cat?#2024-03-2815:46ambrosebsOTOH it's a pretty simple rule to remember that :-> always expands to a :cat no matter what.#2024-03-2815:46ambrosebs(which was the motivation I had, say if you give :-> 1 arg which is a regex, you don't need to :cat )#2024-03-2816:01ambrosebshere's a prototype :-> that supports a :fn property and always wraps args in :cathttps://github.com/metosin/malli/pull/1027/files#2024-03-2816:04ambrosebsI think perhaps allowing non-cat args in :=> can be done separately (if ever) and we keep :-> consistently wrapping :cat.#2024-03-2816:11ambrosebsI'm not a big fan of :-> without children. It starts to look like in an infix op. [:-> :a :-> :b]#2024-03-2820:34pezI’m holding malli wrong again. The schema [:merge :map :map] works https://malli.io/?value=%7B%3Aa%2042%7D&schema=%5B%3Amerge%20%3Amap%20%3Amap%5D, but not in my repl.
(comment
(require '[malli.core :as m])
(m/validate [:merge :map :map] {:name "World"})
;=> #error {:message ":malli.core/invalid- … ema {:schema :merge}" , :data {:type :malli.core/invalid-schema, :data {:schema :merge}}} {:type :repl/invoke, :name "<eval>", :js "malli.core.validate. … 77),"World"], null))" , :source "(m/validate [:merge … ap] {:name "World"})" , :source-map-json "{"version":3,↵ "file … name \"World\"})"]}↵" , …}
:rcf)
Ah, I see it now, I need to merge the malli.util schemas. I guess it could be made clearer on the playground which schemas are merged?#2024-03-2908:53ikitommiIndeed it would be, also, it's using really old version of malli. And the code is copy&paste from Borkdudes really old sci-playground. All improvements most welcome!#2024-03-3018:19ambrosebsI fixed a bug in the :map-of generator and improved all gen/such-that failures to print the schema that failed https://github.com/metosin/malli/pull/1029#2024-03-3018:37ambrosebsof course I immediately found another bug. I will add the fixes.#2024-03-3019:00ambrosebsMake that 3 bug fixes and 1 enhancement.#2024-03-3109:12Adham OmranHey all, I again need help understanding why I'm having this error, according to the documentation :* is used for repetition, which works in the normal case (last 2 lines) but in a function schema it does not work as it should and tells me my input needs to be a map when it should be a vector of maps, what is the problem?#2024-03-3109:22radhikaargs:`[:cat [:* map?] :string]` can be interpreted infix as args:_[map?* string]_
for example `[{} {} {} "foo"]` would be valid args.
i.e. args must be sequence beginning with any number of maps and ending with exactly one string value.
it seems you intended args:`[:cat [:sequential map?] :string]` instead#2024-03-3109:233starblazeThe :cat inside your function schema is the argument list. So right now your schema accepts variable argument count where last item is a string.#2024-03-3109:27Adham Omran> it seems you intended args:`[:cat [:sequential map?] :string]` instead
It seems like that, I must have misunderstood the documentation. I'll try this and see how it works.#2024-03-3109:283starblazeIs it actually the correct solution? I think that the proposed schema cannot match (test-fn "string")#2024-03-3109:28Adham Omran> accepts variable argument count
So if I understand that correctly, a variable map argument count, so any number of maps and the last argument is a string.#2024-03-3109:293starblazeThis might be the schema (haven't checked) [:=> [:cat [:tuple [:cat [:* :map]] :string]] :any]#2024-03-3109:30Adham Omran> Is it actually the correct solution?
In this case it should be since my input must have at least one map in a vector
(test-fn [{}] "") is what I want to pass.#2024-03-3109:313starblazedoes the schema accept (test-fn [] "")?#2024-03-3109:343starblazeAnyways, to understand the function schema better imagine it like this [:=> args-schema return-schema]
So if I have something like [=> [:cat :string :int] :boolean] and call it with (the-fn "foo" 3) then it checks if ["foo" 3] conforms [:cat :string :int] and then checks the return value with return-schema.#2024-03-3109:37Adham OmranI see, thank you!#2024-04-0100:03PanelIs there a way to defined map-entry in a registry ?
I want to set properties at the entry level .
This is possible:
{:x [int? {:title "X parameter"
:description "Description for X parameter"
:json-schema/default 42}]
But i can't find a way to do:
[:x {:title "X parameter"
:description "Description for X parameter"
:json-schema/default 42}
int?]#2024-04-0109:58roklenarcicI don’t see instance-of? in standard schemas… how do I declare instance of type as schema?#2024-04-0114:27escherizehttps://github.com/metabase/metabase/blob/8a9a0a162bdc7c992d059175fd9d5df9784a1d51/src/metabase/util/malli/schema.clj#L37#2024-04-0116:29ambrosebsalso https://github.com/metosin/malli/issues/1007#issuecomment-1925620537#2024-04-0118:36roklenarcicOh so I just put class symbol there?#2024-04-0204:43ambrosebsHaving some very robust results adding constraints to schemas. https://github.com/metosin/malli/pull/1025
In particular, because the constraints are inside the schemas, generators are much more specific. e.g., the generator for [:and int? [:> 739] [:< 741]] almost always fails but this always succeeds.
(is (= 740 (mg/generate [:int {:> 739 :< 741}])))#2024-04-0205:20ambrosebsI haven't implemented the generators yet for this, but another interesting example:
(is (= ["should be distinct: 3 provided 2 times"
"should be sorted: index 1 has 3 but expected 2"]
(me/humanize (m/explain
[:sequential {:sorted true
:distinct true} :any]
[1 3 3 2]))))#2024-04-0205:43ambrosebsTo give some idea of the improved generator expressivity, this is another way of writing the schema in the OP:
(is (= 740 (mg/generate [:int {:and [[:not [:<= 739]]
[:not [:>= 741]]]}])))
#2024-04-0213:14Ben SlessHow's that different from min and max properties?#2024-04-0213:15Ben SlessFor other types that's pretty neat#2024-04-0217:23ambrosebssuperficially, min/max is inclusive (<=/>=). But on a fundamental level, this is no different than properties. This is a propositional logic where the atomic propositions are things like min/max/sorted/distinct/</>.#2024-04-0217:24ambrosebsIt would be trivial to add </> support in the same way as min/max, but I think this abstracts over that entire idea.#2024-04-0217:24Ben SlessWhich means it has algebra, neat#2024-04-0217:24ambrosebsI thought you would enjoy 🙂#2024-04-0217:25Ben Slessbtw, if you're doing strings, it's worth it to add some caregory predicates, such as alpha, alnum, numeric, etc#2024-04-0217:26ambrosebsoh yeah, and there's already generators for that. that would work nicely.#2024-04-0217:27Ben SlessExactly, and it can be implemented efficiently using https://docs.oracle.com/javase/8/docs/api/java/lang/Character.html predicates#2024-04-0217:28Ben SlessThen just for fun you could add :re and we could get rid of regex schemas which aren't comparable because java regexes don't have value semantics (boo)#2024-04-0217:28ambrosebsoh, and test.chuck has a re string generator.#2024-04-0217:29ambrosebsis there a generator for :re?#2024-04-0217:30Ben SlessIt uses test.chuck if it's on the class path#2024-04-0217:30Ben SlessAnd sorry to make more work for you but don't forget time schemas#2024-04-0217:30ambrosebselaborate your ideas#2024-04-0217:31ambrosebsI want to really test this abstraction.#2024-04-0217:32Ben SlessI mean these, I worked really hard on that sh*t would be a shame if they don't get to enjoy the new constraints features
https://github.com/metosin/malli/blob/master/src/malli/experimental/time.cljc#2024-04-0217:32ambrosebsah, and which constraints are you thinking?#2024-04-0217:32ambrosebslike before/after?#2024-04-0217:33Ben SlessYes. They already support min and max, so mostly make sure they aren't left out of the party#2024-04-0217:34ambrosebsNice. are min/max inclusive here?#2024-04-0217:39Ben SlessI think so#2024-04-0217:39Ben SlessCheck line 30#2024-04-0217:42ambrosebsand there's no generators currently hooked up?#2024-04-0217:52Ben SlessThere are.
Getting them right was a pain#2024-04-0217:53Ben SlessHere
https://github.com/metosin/malli/blob/master/src/malli/experimental/time/generator.cljc#2024-04-0217:54ambrosebsnice. yes I've been there.#2024-04-0217:55ambrosebsThis propositional logic might get us out of :fn guard hell. pretending we don't have nat-int? for a second, this could be the schema for nth:
[:=> {:refine [v i]
[:and
[:<= 0 i]
[:< i [:count v]]]}
[:cat :vector :int]
:any]
#2024-04-0217:57ambrosebsI have no idea what implications that has but just an idea that popped up.#2024-04-0217:59Ben SlessThis is starting to look like a where clause in datomic 🙂#2024-04-0218:00Ben SlessThough it loses some information, the return type isn't any, it's a function of the vector index#2024-04-0218:13ambrosebsright, but I'm still working on polymorphic schemas 🙂#2024-04-0218:14ambrosebsIf I'm unabashedly wishfully thinking:
(m/all [x]
[:=> {:and (m/refine [v i]
[:and
[:<= 0 i]
[:< i [:count v]]])}
[:cat [:vector x] :int]
x])
#2024-04-0218:17Ben SlessThat doesn't handle a heterogeneous vector, right?
Still, I'd buy it#2024-04-0218:21ambrosebsmight be more like:
(m/all [x :*, n :- nat-int?]
[:=>
[:cat [:tuple [:.. x]] [:= n]]
[:..nth x n]])#2024-04-0218:21ambrosebswhere [:..] is regex that expands at "instantiation time" of the all.#2024-04-0218:23ambrosebs[:inst ::nth [:cat :a :b :c] 2]
=>
[:=> [:cat [:tuple :a :b :c] [:= 2]] :c]
#2024-04-0218:24ambrosebsand the [:nth.. schema implicitly relates x and n?#2024-04-0218:27ambrosebsthere's a bunch of details I'm working out to get something like this, but I think we can look to type theory for inspiration on how to create an expressive specification language.#2024-04-0218:29ambrosebsthe main caveat is that you can't really instrument a polymorphic fn, so you'd need to compile a monomorphic one for instrumentation, and just use the polymorphic one for generation. Still working on how to do that. like, one detail here is that if you instantiate n to nat-int to generate the most general version of the schema for instrumentation purposes, you get [:= nat-int] as the second arg. Obviously not what you want. Perhaps n is really a spec here instead of a nat. Stuff like that.#2024-04-0218:34ambrosebsWould be curious to know what this would look like starting from your observation that everything resembles datalog, but I don't know how to create verification system from that foundation so that's for someone else to explore 🙂#2024-04-0218:35ambrosebsmy formal training in type theory is leaking through exposing my biases/limitations. it's my one trick.#2024-04-0218:52ambrosebsfleshed it out a bit, I think this should check length bounds for instrumentation and additionally generators of this schema will know how to generate good returns.
(m/all [x :*, n :< nat-int?]
[:=> {:and (m/refine {{:keys [v i]} :args}
[:< i [:count v]])}
[:catn
[:v [:schema x]]
[:i n]]
[:..nth x n]])
[:inst ::nth [:cat :a :b :c] [:= 2]]
;=> [:=> {:and ..} [:cat [:schema [:cat :a :b :c]] [:= 2]] :c]
[:inst ::nth [:* :int] nat-int?]
;=> [:=> {:and ..} [:cat [:schema [:* :int]] nat-int?] :int]
;; for instrumentation:
[:inst ::nth [:* :any] nat-int?]
;=> [:=> {:and ..} [:cat [:schema [:* :any]] nat-int?] :any]#2024-04-0218:55ambrosebsThe trick is that [:..nth [:* :int] nat-int?] => :int and [:..nth [:cat :a :b] [:= 0]] => :a. And probably [:..nth [:cat :a :b] nat-int?]=>[:or :a :b].#2024-04-0219:14Ben SlessWhy aren't you speccing it like a map?
I guess heterogeneous and homogeneous vectors could be specced differently#2024-04-0219:18ambrosebsthis was my only idea, what were you thinking?#2024-04-0219:21Ben SlessToo tired to be coherent, but just however you'll describe a map if the vector is heterogeneous. A homogeneous vector could be defined like you have now. The definitions should overlap for homogeneous vector, too#2024-04-0302:36millettjonI like where this is heading.
It is much cleaner to have a :map with constraints in the properties versus having to nest it inside :and or :fn etc.
The symmetry with the actual shape is maintained.#2024-04-0302:39millettjonFor instance, I have been working on some visualizations of schemas and it requires walking the schema tree and removing all the nodes that are really constraints since the`:maps` are the bits I am interested in.#2024-04-0303:08Ben SlessSuggestion: use the format property like json schema does. Thinking forward to things like email, IP, etc#2024-04-0304:22ambrosebs@UK0810AQ2 are you referring to the :format field in malli.json-schema? I don't follow.#2024-04-0304:30Ben SlessFor strings, instead of :alpha true, :format :alpha#2024-04-0304:30Ben SlessA single key to look at#2024-04-0304:45ambrosebsIt's just sugar for an atomic proposition: {prop true} => {:and [[prop]]}.
{:alpha true :numeric true :max 10} => {:and [[:alpha] [:numeric] [:max 10]]}#2024-04-0304:47ambrosebsYou can even do {:gen/alpha true, :alphanumeric true} to generate alpha in generators and alphanumeric in validators.#2024-04-0304:47ambrosebsthough I haven't implemented it, that's how I've set up the syntax.#2024-04-0305:01ambrosebsI've only done the validator side atm. there [:gen/alpha] expands to the top proposition [:any] so it usually has no effect on the proposition during validation.#2024-04-0305:08ambrosebsto give you an idea, these are the propositions I've supported for :string alone, along with their :gen version. they all use the {prop true} syntax:
:max :min :alphanumeric :non-alphanumeric :letters :non-letters :numeric :non-numeric :alpha :non-alpha :sorted :distinct :palindrome :trim :triml :trimr :trim-newline :blank :non-blank :escapes :includes
#2024-04-0305:08ambrosebsit's easily extensible#2024-04-0305:10ambrosebswell, {:includes "foo"} and
{:escapes {\- "_MINUS_"}}
actually make use of their val.#2024-04-0305:11ambrosebsand min/max of course.#2024-04-0305:24Ben SlessWhat threw me in a loop is that (and alpha numeric) is an empty set#2024-04-0305:26Ben SlessThe sugar is confusion imo#2024-04-0305:46ambrosebs{:max 10 :or [[:alpha] [:numeric]]} rather#2024-04-0305:47ambrosebs{:distinct true :sorted true :max 10} is a better example of a conjunction.#2024-04-0305:48ambrosebsThe rule is the same as always: {:min 10 :max 12} is a conjunction. we just have more atomic propositions now.#2024-04-0305:57Ben SlessYes, but the code for handling them would be annoying#2024-04-0305:58Ben Sless{:format [:and :alpha :distinct]} is clearer for the reader and simpler to implement imo#2024-04-0305:58Ben SlessIt's uniform and simple to compile to a bunch of predicates#2024-04-0306:38ambrosebsIt's a dozen lines to desugar in a fully extensible way. Plus I want to save the keyword syntax for [:contains K] sugar like s/keys.
(defn -constraint-from-properties [properties constraint-opts options]
(let [{:keys [flat-property-keys nested-property-keys]} (->constraint-opts constraint-opts)]
(when-some [cs (-> []
(into (keep #(when-some [[_ v] (find properties %)]
(into [%] v)))
nested-property-keys)
(into (keep #(when-some [[_ v] (find properties %)]
(conj [%] v)))
flat-property-keys)
not-empty)]
(if (= 1 (count cs))
(first cs)
(into [:and] cs)))))#2024-04-0306:38ambrosebs[:and :x :y] instead of [:and [:contains :x] [:contains :y]].#2024-04-0306:41ambrosebsAt least for schemas that support contains?. Haven't decided whether to allow unwrapped keywords in other schemas yet.#2024-04-0306:48ambrosebsLet's say I do allow unwrapped keyword in say :string . I think {:and [:alpha :distinct]} is clearer than {:format [:and :alpha :distinct]}.#2024-04-0306:51ambrosebsmainly because "format" isn't a proposition.#2024-04-0306:53ambrosebsIt's unclear how far this concept can go, but I want the user to think property == proposition. And property map == conjunction.#2024-04-0306:54Ben Slessyeah, format isn't a proposition, it's more like a predicate#2024-04-0306:55ambrosebsearlier I renamed :and to things like :keyset :keys . I think once I saw the abstraction of everything is a proposition, the current philosophy started to make sense to me.#2024-04-0306:56ambrosebsevery single schema can support :and , and it will make sense.#2024-04-0306:57ambrosebsI think :format for :string is similar to my thinking of :keyset constraints for :map. We can abstract over the concept of "thing is true for schema".#2024-04-0306:58Ben Slessmy thinking about it is tainted by how json schema does it#2024-04-0306:59ambrosebsyeah and my thinking about properties was tainted by min/max. It was tailored for each schema in different ways and didn't seem like a coherent abstraction beyond syntax.#2024-04-0306:59ambrosebsbut that was my thinking too: a property is something tied to the schema type.#2024-04-0306:59ambrosebsuntil like yesterday. so it's a new idea.#2024-04-0307:05ambrosebshere's a good existing example of using true.
(mg/generate
[:double {:gen/infinite? true, :gen/NaN? true}]
{:seed 1})
They are both atomic propositions. could use them like {:gen/xor [:gen/infinite? :gen/NaN?]} to never mix ##Inf and NaN in the same run (in theory, let's see when the rubber meets the road).#2024-04-0215:57Oliver MarksI am using the malli reitit ring middleware all is working but there is one thing I am curious about can you adjust the humanized response to include keys.
"humanized":["invalid type","invalid type"]
In the above example it tells me there are 2 errors but it does not include the key in the response so I don't know the field that had the error, Is there a way to add in this information or map it to the submitted data so that I can put an error marker on the form fields on the frontend, wanted to check before I role some kind of work around.
This sounds like it may do the job but the example is for manually validation so not sure if I can modify the ring reitit middleware in the same way ?
https://github.com/metosin/malli/blob/master/docs/tips.md#getting-error-values-into-humanized-result#2024-04-0316:48RaviHi I am trying to use malli to validate the input LocalDate string in params
I did try to use malli.experimental.time but seems like it cannot handle the localDate string
Ideally i used regex ealier but this breaks the Swagger doc as json is not able to convert regex. So if someone has some experiance with this that will be helpful
(def some-obj
[:map
[:rate [:maybe number?]]
[:myDate [:maybe [:re #"^\d{4}-\d{2}-\d{2}"]]]
[:myDate2 [:maybe [:re #"^\d{4}-\d{2}-\d{2}"]]]])
Want to validate localDate string like "2020-12-02"
Swagger Error
Caused by: com.fasterxml.jackson.core.JsonGenerationException: Cannot JSON encode object of class: class java.util.regex.Pattern: ^\d{4}-\d{2}-\d{2}
#2024-04-0316:53delaguardohave you tried :time/local-date?#2024-04-0316:56delaguardoit needs decoder https://github.com/metosin/malli/blob/master/test/malli/experimental/time/transform_test.cljc#L13
but should work#2024-04-0316:59delaguardoin case you are using reitit there is a doc how to configure it
https://cljdoc.org/d/metosin/reitit/0.7.0-alpha7/doc/coercion/malli#configuring-coercion#2024-04-0613:42frozenlockIs there a library to generate hiccup forms from malli? I'm looking for something on the server side.#2024-04-0615:06ambrosebsthere's a schema here, try passing it to malli.generator/generate https://github.com/metosin/malli?tab=readme-ov-file#parsing-values#2024-04-0615:22frozenlockI'm not sure I understand how it would work. I meant, for example, to generate:
[:input {:type :text
:required true
:value this}]
from
[:string]
#2024-04-0706:01delaguardothis might help
https://flexiana.com/2022/02/malli-and-html-forms-as-data#2024-04-0707:03Игорь ЛисовцовI recently came across the same question. I did not find suitable ready-made solutions for generating forms, so I decided to do everything myself. To generate forms based on schemas, I added additional properties to the schema. Since I plan to use schemas for both the database and the forms, it was necessary to separate the schemas of the form and the basic schema.#2024-04-0707:06Игорь ЛисовцовSo, for now, I've created simple functions that generate forms based on malli's type, like so#2024-04-0817:03escherizeCool! @U02TPTEM5TJ, is that open source?#2024-04-0915:20frozenlockInteresting, thanks for sharing your approach!#2024-04-0920:52Игорь Лисовцов@U051GFP2V ofc, feel free to use :)#2024-04-1006:51Игорь Лисовцовhttps://github.com/IngvarListard/medical-card/blob/main/src/medical_card/ui/components/inputs.clj#2024-04-0814:56Oliver MarksUsing malli to pass http requests can you have a sequence of query parameters, someone else asked here https://clojurians.slack.com/archives/CLDK6MFMK/p1637852630157700 with no answer, seems swagger is happy in that you can add them to query params as comma separated values but when the request is made malli return's invalid type on the vector, should this work ? I am seeing the same behavior as @haiyuan.vinurs#2024-04-1003:11ambrosebsNot sure about the swagger and parsing part, but ring middleware usually converts duplicate query params into collections:
(let [app (ring/ring-handler
(ring/router
["/ping" {:get #(select-keys % [:params :query-params])}]
{:data {:middleware [parameters/parameters-middleware]}}))]
(is (= {:query-params {"kikka" "0"}
:params {"kikka" "0"}}
(app {:request-method :get
:uri "/ping"
:query-string "kikka=0"})))
(is (= {:query-params {"kikka" ["0" "1"]}
:params {"kikka" ["0" "1"]}}
(app {:request-method :get
:uri "/ping"
:query-string "kikka=0&kikka=1"}))))
#2024-04-1003:12ambrosebsFWIW there's a related issue about spec + parsing here: https://github.com/metosin/reitit/issues/298#2024-04-1008:42Oliver Marksnice find I notice when testing in swagger it actually setup the query as ?key=value1,value2 so its comma separating perhaps I will add that to the issue, as the above shows & as a separator#2024-04-0921:01Игорь ЛисовцовIs there any decent way to update entry's property? For example, I want to update name Event's property but save schema, but there is no update-property-in or something function
Thanks.#2024-04-1002:28ambrosebsPerhaps https://github.com/metosin/malli/blob/2bdc1cf21705bca50f7558c6fd153b06df573f78/src/malli/core.cljc#L349-L350#2024-04-1006:53Игорь ЛисовцовThank you. I'll try again to dig about the function today. But yesterday it didn't work for my use case#2024-04-1014:24Игорь ЛисовцовUnfortunately, doesn't work with map entries#2024-04-1015:52Игорь ЛисовцовSolved it like below#2024-04-1016:53ambrosebsThis is the closest I could get to updating the properties, but it's pretty similar to your solution:
(m/-set (m/schema [:map [:me {:a 1} :int]])
[:me {:a 2}] :int)
;=> [:map [:me {:a 2} :int]]
#2024-04-1017:13ambrosebs(defn update-entry-properties [s k f & args]
(m/from-ast
(update-in (m/ast s)
[:keys k :properties] #(apply f % args))))
(update-entry-properties
[:map [:me {:a 1} :int]]
:me
assoc :b 1)
;=> [:map [:me {:a 1, :b 1} :int]]
#2024-04-1017:24ambrosebsThis is probably the proper implementation, works with anything that has entries AFAICT:
(defn update-entry-properties [s k f & args]
(let [s (m/schema s)
e (or (mu/find s k)
(m/-fail! ::no-entry {:schema s :k k}))
[k p v] (case (count e)
2 [(first e) nil (second e)]
3 e)]
(m/-set-entries s [k (apply f p args)] v)))
(is (= [:map [:me {:a 1, :b 1} :int]]
(m/form
(update-entry-properties
[:map [:me {:a 1} :int]]
:me
assoc :b 1))))
(is (= [:orn [:me {:a 1 :b 1} :int]]
(m/form
(update-entry-properties
[:orn [:me {:a 1} :int]]
:me
assoc :b 1))))
#2024-04-1017:32ambrosebshttps://github.com/metosin/malli/pull/1037#2024-04-1018:26Игорь Лисовцовwow! thank you a lot! Looks great!#2024-04-1405:23ambrosebsProposing a new syntax for functions, and making function schemas extensible via protocols https://github.com/metosin/malli/pull/1027
[:function [:=> [:cat] :int] [:=> [:cat :int] :int]]
==
[:ifn [:-> :int] [:-> :int :int]]
#2024-04-1502:48ambrosebsPretty neat use of a malli schema proxy here to support a regal schema as a wrapper of :re https://github.com/lambdaisland/regal/pull/48/files#2024-04-2006:34ikitommiindeed! Once the JSON Schema -> Malli is complete, you could do the interop like:
[:malli.json-schema/schema
{"$schema": "",
"$id": "",
"title": "Product",
"description": "A product from Acme's catalog",
"type": "object",
"properties": {
"productId": {
"description": "The unique identifier for a product",
"type": "integer"
}
}]#2024-04-1515:08roklenarcicIs there a way to make a better humanized output when fn schema fails?
Value:
[#object["[]"] 1000M]
Errors:
["unknown error"]
Schema:
[:and
[:catn
[:q :roklenarcic.trade.fifo/fifo-queue]
[:quantity [:maybe decimal?]]]
[:fn
#object["roklenarcic.trade.fifo$eval19078$take_positions__19079$fn__19082@55530ef1"]]]
It just says “unknown error”. And need to guess that it’s caused by the :fn schema and after figuring that one out I need to find which predicate this is…#2024-04-1515:11Noah Bogarthttps://github.com/metosin/malli/?tab=readme-ov-file#custom-error-messages#2024-04-1515:12Noah Bogarteither :error/message or :error/fn will be what you need here#2024-04-1515:17roklenarcichm… I was looking more in the direction of SCI#2024-04-1515:17roklenarcicso I can see function definition#2024-04-1515:23roklenarcicHm yes, this is much better:
Schema:
[:catn
[:q
[:and
:roklenarcic.trade.fifo/fifo-queue
[:fn
(fn
[q]
(apply
=
(mapv
(comp pos? :roklenarcic.trade/quantity)
q)))]]]
[:quantity [:maybe decimal?]]]
But that printout is atrocious#2024-04-1515:25Noah Bogartright, so write [:fn {:error/fn (fn [{:keys [value]} _] (str "given " value ", expected something else"))} (fn [q] ...)]#2024-04-1515:27roklenarcicI cannot, because I just provide the function, another library wraps it into :fn#2024-04-1515:27Noah Bogartthen there's nothing you can do here#2024-04-1515:27Noah Bogartwhat library are you using?#2024-04-1515:27roklenarcicyeah, it seems#2024-04-1515:27roklenarcicGuardrails#2024-04-1515:28Noah Bogartcan you share the calling code here? aka the code you're calling to make this schema#2024-04-1515:28roklenarcicIt’s a defn macro wrapper:
(>defn move-positions
"Move from one fifo queue to another while scaling as appropriate"
[fifo-map {::keys [factor pre-quantity post-quantity new-security] :as x}]
[::fifo/fifo-map ::<> => [:map-of ::security/id ::trade/lot-seq]]
#2024-04-1515:29roklenarcicsorry wrong def#2024-04-1515:29roklenarcicIn any case I need to rethink my approach#2024-04-1515:29roklenarcicthank you#2024-04-1515:33Noah Bogartguardrails takes malli schemas, so you can say [:and ::fifo/fifo-queue [:fn {:error/fn (fn [{:keys [value]} _] ...)} (fn [q] (your fn here))]] [:maybe decimal?] or move that :and to a helper schema: (def helper (m/schema [:and ::fifo/fifo-queue [:fn {:error/fn ...} (fn ...)]])) and then in your defn call:
(def helper
(m/schema
[:and ::fifo/fifo-queue
[:fn {:error/fn ...}
(fn ...)]]))
(>defn cool-func
"docs"
[q quantity]
[helper [:maybe decimal?]]
...)#2024-04-1515:34roklenarcicyeah I can make it liek this, but I was hoping to use their such that mechanism#2024-04-1515:34Noah Bogartoh, i forgot about that, hmm#2024-04-1515:35roklenarcicno I mean that’s fine#2024-04-1515:35Noah Bogartmaybe ask in #C68M60S4F or in the guardrails issues, i bet they can help.#2024-04-1515:43roklenarcicYeah thanks #2024-04-1515:47Noah Bogarthttps://github.com/fulcrologic/guardrails/blob/2ee77d1d7672d54ff0be251f54a18f5e703a7d16/src/main/com/fulcrologic/guardrails/core.cljc#L505-L508#2024-04-1515:47Noah Bogartthere you go, that's the set of lines where the such that is wrapped in [:fn]#2024-04-1621:38Nikolas PafitisHow can i use malli to validate if the value of a map entry is a valid malli schema?#2024-04-2007:55ikitommiyou can call (m/schema value) on it at least:
(def Schema [:fn {:error/fn
(fn [{:keys [value]} _]
(str value " is not a valid Schema"))}
(m/-safe-pred m/schema)])
(->> :int (m/explain Schema) (me/humanize))
; => nil
(->> :intz (m/explain Schema) (me/humanize))
; => [":intz is not a valid Schema"]#2024-04-1621:39Nikolas PafitisSpecifically i want to check if the form is a valid map schema form#2024-04-1621:55crimeministerI am building a schema based on an external standard that makes a lot of use of inheritance. Certain objects inherit from parent definitions, optionally adding additional attributes. By preference I am using a local registry ("the data way") and would like to be able to capture these relationships in the registry definitions. Naively I imagined I could do something similar to this simplified example:
(def MySchema
[:schema {:registry {::parent [:map
[:x string?]
[:y number?]]
::child [:merge
[:ref ::parent]
[:map
[:z boolean?]]]}}
[:map
[:foo ::parent]
[:bar ::child]]])
It seems as though I can't use :merge within the registry, however. I expect I could extract these definitions into separate vars and combine them with (merge) , but perhaps I am missing a more idiomatic approach? Any advice appreciated.#2024-04-2007:56ikitommiI have an experimental branch with map-like schema supporting :extends and :abstract.#2024-04-2007:57ikitommiwill polish that at some point and push out for comments.#2024-04-2007:59ikitommialso, you can use :merge with registry, just have to ensure :merge etc. utility schemas are available:
;; add utility schema to global registry
(mr/set-default-registry!
(mr/composite-registry
(m/default-schemas)
(mu/schemas)))
(def MySchema
(m/schema
[:schema {:registry {::parent [:map
[:x string?]
[:y number?]]
::child [:merge
[:ref ::parent]
[:map
[:z boolean?]]]}}
[:map
[:foo ::parent]
[:bar ::child]]]))
(mg/sample MySchema)
;({:foo {:x "", :y 2.0}, :bar {:x "", :y -1.0, :z true}}
; {:foo {:x "7", :y -1.0}, :bar {:x "X", :y -0.5, :z true}}
; {:foo {:x "", :y -1}, :bar {:x "", :y 2.0, :z false}}
; {:foo {:x "K", :y 0.75}, :bar {:x "w6", :y 0, :z false}}
; {:foo {:x "", :y 2.5}, :bar {:x "T", :y -1.125, :z false}}
; {:foo {:x "y", :y -11}, :bar {:x "po4", :y 0, :z false}}
; {:foo {:x "0n", :y 1.0}, :bar {:x "36", :y -2.0, :z true}}
; {:foo {:x "8xc", :y -0.875}, :bar {:x "", :y -1.40625, :z false}}
; {:foo {:x "zwSC", :y 51}, :bar {:x "2bHg50G", :y 0.25, :z false}}
; {:foo {:x "oS5", :y 92}, :bar {:x "75rj74", :y -0.453125, :z false}})#2024-04-2007:59ikitommidoes that look correct?#2024-04-2023:41crimeministerThanks for the info @U055NJ5CC! I'll give this a closer look but at a glance it looks correct.#2024-04-2007:55ikitommiyou can call (m/schema value) on it at least:
(def Schema [:fn {:error/fn
(fn [{:keys [value]} _]
(str value " is not a valid Schema"))}
(m/-safe-pred m/schema)])
(->> :int (m/explain Schema) (me/humanize))
; => nil
(->> :intz (m/explain Schema) (me/humanize))
; => [":intz is not a valid Schema"]#2024-04-1711:24roklenarcicHow do I specify infinite sequence of a type#2024-04-1715:14Noah Bogart:any probably, i don't think there's a function that checks if a sequence is infinite.#2024-04-1715:19roklenarcicno I don’t really want to check if it’s infinite I just don’t want to check all items#2024-04-1715:19roklenarcicI just want a non-exhaustive check#2024-04-1715:28Noah Bogartyou can check a bounded number with :fn, something like [:fn (fn [coll] (m/validate [:sequential :int] (take 5 coll)))] which will only check the first 5 items of the collection#2024-04-1715:31roklenarcicSpec has this feature#2024-04-1715:31roklenarcicevery#2024-04-1718:00ambrosebsmalli is missing this kind of schema. relatedly it also doesn't have a schema that can verify a seqable? of X. This is one area that spec has a leg up over malli and plumatic schema.#2024-04-1718:03Noah Bogarthow does spec handle it?#2024-04-1718:04ambrosebsit has a spec called every that checks up to some number of elements (around 200 iirc). kind of a bounded-count schema.#2024-04-1718:06ambrosebsapparently doesn't conform its elements either. I guess that makes sense.#2024-04-1718:47ambrosebs@U66G3SGP5 could you try this branch? see the tests for usage: https://github.com/metosin/malli/pull/1041#2024-04-1718:48ambrosebshmm cljs tests failed, let me fix that.#2024-04-1711:24roklenarcic[:sequential will validate all and hang#2024-04-1801:45ambrosebsDeterministic pure & impure function generators https://github.com/metosin/malli/pull/1042#2024-04-2013:21Ben SlessMay be a silly question regarding https://github.com/metosin/malli/pull/1044
Why recursively deref query parameters schema rather than pass by reference?
Maybe it's nonsensical to pass an object as a query, but you can still use a schema path into the reference to point out the specific field.
Does it mean that any reference will be inlined in query parameters? Isn't it too aggressive? Doesn't it miss on the point of naming attributes?#2024-04-2013:24ikitommiThat's only for swagger, right?#2024-04-2013:25ikitommiHow does that work with openapi3 / json-schema?#2024-04-2013:25Ben SlessNot sure, I think openapi falls back on swagger which falls back on json schema#2024-04-2013:29ikitommiOpenApi3+ allows non-body params via references, I think swagger(2) doesn't.#2024-04-2013:30Ben Slessah#2024-04-2013:31ikitomminot sure how this works now with openapi3 thou. It would be great to be able to sunset some of the older things (like swagger2-support), many bugs need fixing on both, spanning over 3+ libs.#2024-04-2013:33Ben SlessI see that openapi docs eventually use json-schema, while it should use something that overrides swagger#2024-04-2013:33Ben SlessThree Emacs buffers open for that one 🙃#2024-04-2013:44ikitommijust tested reitit with the latest malli, there is a issue with openapi + definitions: https://github.com/metosin/reitit/pull/675#2024-04-2013:45ikitommiFAIL in (recursive-test) (openapi_test.clj:859)
spec is valid
expected: (nil? (validate spec))
actual: (not (nil? {"errors" "Can't resolve #/definitions/friend", "valid" false}))#2024-04-2013:46ikitommiwait what, it works on github, just fails locally :thinking_face:#2024-04-2013:51ikitommiok, I think it’s this one, https://github.com/metosin/reitit/issues/662. will check with on monday#2024-04-2013:34ikitommiFresh from the oven, [metosin/malli "0.16.0"]! dropped support for Java8, otherwise it’s a non-breaking / patch release. Thanks to all contributors, mainly @ambrosebs 🙇
> • BREAKING: minimum Java-version is now Java11
> • allow changing prefix of json-schema $refs via option :malli.json-schema/definitions-path https://github.com/metosin/malli/pull/1045
> • Inline refs in non-`:body` swagger parameters https://github.com/metosin/malli/pull/1044
> • Fix flaky test https://github.com/metosin/malli/pull/1040
> • Utility to update entry properties: mu/update-entry-properties https://github.com/metosin/malli/pull/1037
> • Fix actions cache https://github.com/metosin/malli/pull/1036
> • Only humanize one of :min / :max when different https://github.com/metosin/malli/pull/1032
> • Distinguish between symbols and strings in humanize https://github.com/metosin/malli/pull/1031
> • Fix :map-of :min and unreachable generator, explain such-that failures https://github.com/metosin/malli/pull/1029#2024-04-2214:36mikerodWhy is the min Java version changing? I didn’t see any rationale in the repo and couldn’t find a commit since then that seemed to have any reason to need Java 11 min.#2024-04-3007:42ikitommiValid point. We discussed this internally and decided to roll-back to support 8. Thanks!#2024-04-2015:47George KiersteinHi!
Beginner question - I'm looking for an equivalent to [:or [.... . Basically I am trying to construct a schema for a map with keys that will be disjoint sets depending on the state of the system - A logical or.
i.e.
Case A: {:a :v, :b :v}
Case B: {:c :v}
I see an [:and ... but not an [:or ... - What would a schema like this look like?
Thanks!#2024-04-2109:44ikitommiHi! If to search this channel, there is a lot of discussion about the keysets. Nothing really good built-in yet.#2024-04-2016:10George KiersteinUnrelated beginner question: Which part of the documentation should I look into in order to use a predicate in a schema that isn't provided by default? I am using https://github.com/clj-commons/manifold (although its a generic question) which provides predicates I would like to use in a schema:
[:map [:source m/stream?]]
The schema above fails with:
Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
:malli.core/invalid-schema
Thanks!#2024-04-2017:22respatializedhttps://github.com/metosin/malli/blob/master/docs/function-schemas.md
function schemas (`[:fn m/stream?]`) are what you’re looking for.#2024-04-2017:31George KiersteinAwesome thanks!#2024-04-2109:48ikitommiWhat if malli had a custom edn-file (`malli.edn`) that was used for all project-spesific config, e.g. read at malli.core startup? Would allow better reuse (between projects) of good configs (registry, custom errors, sci-settings etc). Starting from something like this:
{:dev-mode true
:registry (malli.registry/composite-registry
(malli.core/default-schemas)
(malli.experimental.time/schemas)
(malli.util/schemas))}
#2024-04-2109:56ikitommimalli.edn
{:dev-mode true
:registry (malli.registry/composite-registry
(malli.core/default-schemas)
(malli.experimental.time/schemas)
(malli.util/schemas))}
user.clj
(ns user
(:require [clojure.walk :as walk]
[clojure.edn :as edn]
[malli.core :as m]
[malli.registry :as mr]))
(defn -walk-requiring-resolve [x]
(walk/prewalk #(cond-> % (qualified-symbol? %) (requiring-resolve)) x))
(defn -read-config []
(-> (slurp "malli.edn")
(edn/read-string)
(-walk-requiring-resolve)
(update :registry eval))) ;; (/_\)
(mr/set-default-registry!
(:registry (-read-config)))
(m/deref-recursive
[:merge
[:map [:x :int]]
[:map [:y :int]]])
; => [:map [:x :int] [:y :int]]#2024-04-2109:58ikitommiMy first thought is that thiswould be good, as all the configs & boilerplate of wiring things together would be in one place -> no need to pass options map with everywhere.#2024-04-2109:59borkdudeI don't know. What about front-end projects?#2024-04-2110:01borkdudeIt sounds like this is thinking from a solution rather than from a clear problem statement. The problem statement could be: Passing a config options map everywhere is a bit annoying, what alternative solutions are there to this?#2024-04-2110:01borkdudeOne alternative could be to set a global atom with global config#2024-04-2110:02borkdudeOne decision criterium could be: works across backend and front-end#2024-04-2110:02borkdude(I'm referring to the spreadsheet stuff that Rich Hickey alluded to in his latest talk)#2024-04-2115:283starblazePersonally I don't like having extra files at project root, it feels like logic is scattered around this way. I like the global atom kind of approach.
Speaking from personal experience, I'm writing a full stack, mostly frontend app. Shadow has its own config and at the beginning it was a good enough approach but then I noticed I started to have some duplication in build declaration logic, so I made my own build script, defined cljs dependencies in deps.edn and I didn't need a separate file for shadow config. This was really nice because I know that I just have to look at deps.edn and my build script and I don't have to worry about jumping through several files.
I think the best approach would be to define some function, say set-config! . Developers only need to handle how this function is called and they have the full freedom to shape their build process as needed.#2024-04-2117:26ikitommithanks. Currently, there is already an atom for the global registry (`malli.registry/registry*`), would be better if the whole options could be swapped. My original issue for this is from 2020: https://github.com/metosin/malli/issues/228. To get best out of this, some options would have to be changed from non-qualified keys to qualified to avoid collisions.#2024-04-2117:28ikitommicurrently, if one want’s to disable sci by default, one has to put ::m/disable-sci via options for all the public functions. Same would be if the https://github.com/metosin/malli/pull/1043 would be merged, e.g. option to disable it.#2024-04-2117:32ikitommiI personally like a file as a easy way to configure this (on top of the options atom replacing the registry atom), as it would also remove the need for extra type and mode arguments:
;; - cljs: :closure-defines {malli.registry/type "custom"}
;; - clj: :jvm-opts ["-Dmalli.registry/type=custom"]#2024-04-2117:34ikitommibut, open to comments and feedback#2024-04-2117:38ikitommie.g. currently, when malli.core loads:
1. by default, it adds all schemas into registry (no DCE for cljs)
2. … unless type or mode are set, in case you can start with empty registry and add just the things you need via mr/set-default-registry! (DCE can be used to remove 90%+ of the code for cljs)
with the optional malli.edn file, malli.core would:
1. load the file (if exists), instantiate the registry defined by it
2. if no file present, just use the default registry (all schemas)
3. no type or mode needed (due to 1 - just include the schemas you need)#2024-04-2117:53ambrosebsone idea is a defn-generating macro, that generates malli's api with default options. No state involved, no sharing issues if you pull another library that also uses malli with default options.
(ns my-apps.malli.core (:require [malli.core :as m])
(def my-default-options {::m/disable-sci true})
(m/create-malli-core-api my-default-options)
; => (defn validate ([v] (validate v my-default-options)) ([v options] (m/validate v options))) (defn explain ...) ... etc...#2024-04-2118:053starblaze@U055NJ5CC As long as there is a nice programmatic API, I think a separate file is a fine approach because that would make the initial global registry setting as smooth as possible.#2024-04-2118:063starblaze@U055XFK8V isn't a macro an overkill? Couldn't malli just read from a dynamic var which you could toggle in namespace level (just like ns macro sets *ns* or how all printing is redirected when you rebind *out*)#2024-04-2118:13ambrosebs@U02S4QZAH61 certainly there's costs, in particular each app has their own api. IMO it's the most robust solution if that's a worthy cost to pay to be impervious to parallelism issues or clashes with other libraries.#2024-04-2118:14ambrosebsjust because code occurs in a namespace doesn't mean it will be evaluated in it either#2024-04-2118:173starblazeoh, so that means that dynamic vars are not parallel-computing friendly?#2024-04-2118:17ambrosebscorrect#2024-04-2118:17ambrosebsthey are thread-local#2024-04-2118:18ambrosebsonce you kick off another thread, you need to convey the bindings to the new thread to keep the same context. many but not all clojure ops do this.#2024-04-2118:18ambrosebsbut my point is more:
(ns ns1)
(defn a [] *ns*)
(ns ns2
(defn b [] (a))
(ns ns3)
(ns2/b) => prints ns3
#2024-04-2118:183starblazeOh yeah, I recently realized that *ns* doesn't refer to the file namespace. so my analogy is wrong.#2024-04-2118:20ambrosebsthe best way to determine which namespace you occur in is with syntax-quote or a namespaced keyword.
(defn this-ns [] (-> ::a namespace symbol))
#2024-04-2118:213starblazethat looks hacky but I appreciate the explanation#2024-04-2118:23ambrosebsit's one of the pillars of clojure's macro system, so a macro can distinguish between the namespace it occurs in and the ns it expands in via syntax-quote.#2024-04-2206:41tlonistthis question might have been answered multiple times, but here it goes.
why does malli schema use vectors over maps when describing its data structure? IMO maps are easier to reason out a model than vector, and also easier to navigate.#2024-04-2207:03Alex SkyThis is true but it is not clear where in maps to add additional information. I would consider vector here as hiccup format.
[:map
[:foo {:option "dditional information"} :string]]#2024-04-2207:18tlonistadditional information can be expressed using functions with parameters maybe?
{(s/optional-key :something) (s/some-operation :something2)}#2024-04-2207:24Alex SkyIn my humble opinion, it looks scary. 😁 I've worked with cljs and hiccup I found it very convenient in this case. You can also use map in malli if you want.
{:foo :string} #2024-04-2207:27Alex Skyhttps://github.com/metosin/malli?tab=readme-ov-file#map-syntax#2024-04-2219:11escherizehttps://github.com/metosin/malli?tab=readme-ov-file#lite might be what you expected, too.#2024-04-3007:37ikitommiYes, Lite is closest what you want I think. Before starting to write Malli, idea was to propose “Plumatic Schema 2”, which would be the fusion of spec and schema, while using the Schema syntax. But it came clear that I would have to rewrite mostly everything to get the features and performance in so decided to start from scratch instead. hiccup was the most compact, had ordering and no ambiguity issues. So, hiccup it was. map and lite are addons to it.#2024-04-3008:08juhoteperiKeeping the property order can also be useful, if not for the datastructures, at least for generated documentation.
Maps would lose the order after ~8 properties and then your Swagger docs would be on random order which might not look as nice.#2024-04-2215:30jmckitrickI’ll cross-post my question here since this is really where it belongs. Simply put, is there a migration path from data-spec to malli that retains compatibility with reitit-swagger?#2024-04-2310:54jmckitrickI’ve been tinkering, and it seems like it will work. I need to try some more complex cases by hand, but so far, so good.#2024-04-2311:05Alex Skywhat is it data-spec ? Could you provide links
upd: data-spec = clojure.spec ? 🙂#2024-04-2311:06jmckitrickNo, just changing the coercion type and using malli vector schema. Swagger seems to understand it.#2024-04-2311:44jmckitrick@UCQTFTW8P I don’t understand your question 🙃#2024-04-2311:59Alex SkyOk, nevermind:slightly_smiling_face:#2024-04-2312:01jmckitrickhttps://cljdoc.org/d/metosin/spec-tools/0.10.6/doc/data-specs#2024-04-3007:39ikitommi@U061KMSM7 reitit understands malli-lite systax oob, so the migration should be relatively streightforward, e.g.
{:id int?, :address {:street string?, :number int?}}
is valid in both malli & data-spec#2024-04-3007:37ikitommiYes, Lite is closest what you want I think. Before starting to write Malli, idea was to propose “Plumatic Schema 2”, which would be the fusion of spec and schema, while using the Schema syntax. But it came clear that I would have to rewrite mostly everything to get the features and performance in so decided to start from scratch instead. hiccup was the most compact, had ordering and no ambiguity issues. So, hiccup it was. map and lite are addons to it.#2024-04-2312:21jmckitrickI have a parameter with malli schema [:alt string? number?] in the body, but in the swagger page it is rendered as ‘object’ or ‘{}’ in the example. How can I fix this?#2024-04-2312:43jmckitrickI cannot get ‘optional’ items to work either. I had them working with data-spec, so it’s probably a small fix.#2024-04-2314:16jmckitrickI got ‘optional’ to work… but not ‘alt’, unless that’s not really supported by swagger#2024-04-2318:15ambrosebsPerhaps try :or instead of :alt if applicable?#2024-04-2419:46rgorrepatiI am new to malli. I am having trouble with generating :time/instant#2024-04-2503:22Ben SlessYou need to require the generators namespace for time schemas
https://github.com/metosin/malli/blob/master/src/malli/experimental/time/generator.cljc#L1#2024-04-2517:07rgorrepatioh! that’s side effecty. It works. Thank you#2024-04-2419:47rgorrepati(mg/generate :time/instant {:registry (met/schemas)}) #2024-04-2419:48rgorrepatireturns a no-generator execution error#2024-04-2512:55jmckitrickI’m currently using data-spec to add swagger info to routes for swagger-ui. We want to migrate to malli . I’m looking at malli.swagger/transform but cannot tell where to integrate it into the route data. Suggestions?#2024-04-2518:17jmckitrick@U055NJ5CC thoughts? ☝️#2024-04-2607:23ikitommiyou can annotate the schemas with :swagger/description etc. Have you looked the https://github.com/metosin/malli?tab=readme-ov-file#swagger2?#2024-04-2610:50jmckitrickI’ve figured most of it out, with just a few details left. How can I override the default ‘inline_model’ labels and replace them with my schema names?#2024-04-2610:50jmckitrick#2024-04-2610:51jmckitrick#2024-04-2512:56jmckitrickRight now the schema is in :post -> :parameters -> :body#2024-04-2513:12vemvIs anyone regularly inspecting/navigating Malli schemas in a tool like REBL, portal, the CIDER inspector, etc?
My question is two-faced
• what exact tool are you using
• what pattern are you using for defining schemas
◦ For me the tricky part is keeping everything defined as plain data (with all the benefits it carries), while keeping names (e.g. the Foo in (def Foo :string))
There are surely plenty of choices (vars, registries, and more), and I wouldn't say it's hard to hack something. But I wonder if anyone is doing something remarkable today#2024-04-2513:14vemvIn CIDER we have a spec browser https://docs.cider.mx/cider/usage/misc_features.html#browsing-the-clojure-spec-registry , as of today we couldn't realistically offer that for Malli I believe as it has a far wider variety of usage patterns#2024-04-2513:25Ben SlessSince malli schemas are Just Data that's often enough for me when inspecting and navigating#2024-04-2513:28vemvNames are lost though, and things can get expanded beyond comprehensibility#2024-04-2514:35Ben SlessTrue, which is why I'm stoked for var ref schemas#2024-04-2514:40vemvIn principle I want to like them, but then doesn't stuff stop being data?
Simple example:
(def Foo
(into NewFoo
[[:id Id]]))
This won't work if NewFoo became #'NewFoo#2024-04-2514:42vemv...there's mu/merge and friends, but those return a reify, i.e. also not-data#2024-04-2514:59Ben Slessoh right, they return an into-schema#2024-04-2515:57Ben SlessI'm spoiled by the inspector that freely digs into closed over fields#2024-04-2516:07vemvYeah it shoulnd't be bad at all, but as have the chance to design some things from scratch I wanted to see if there's an optimal pattern#2024-04-2516:08vemvMost likely I'd go for a keyword registry, and perhaps datafy/nav somewhere in the mix#2024-04-2513:38Tommi MartinHello, would it be possible to get pointed to the syntax of [:multi {:dispatch custom-dispatch} schema?
What is the expected return of the custom-dispatch ?
In examples it is a keyword or a string. but what about cases where the multi is called from inside ::m/default ? in those cases the dispatch function receives a map of all fields that didn't match the schema.
I'm sorry maybe i was blind and couldn't find the answer in documentation.
Code example in a thread:#2024-04-2513:38Tommi Martin(require '[malli.core :as m])
(defn test-dp
[m]
(clojure.pprint/pprint m)
#_"what am I supposed to return?")
(m/validate
[:map
[:name :string]
[:field1 :string]
[::m/default
[:multi {:dispatch test-dp}
["service" [:map-of :keyword [:map
[:type :string]
[:name :string]
[:new-field :string]]]]
["source" [:map-of :keyword [:map
[:type :string]
[:name :string]
[:some-other-field :string]]]]]]]
{:name "test"
:field1 "test-field"
:random1 {:rng {:type "service"
:name "test1"
:new-field "some-value"}}
:random2 {:rng2 {:type "source"
:name "test2"
:some-other-field "some-other-value"}}})#2024-04-2513:38Tommi Martinvalue of m in test-dp with this code:
{:random1
{:rng {:type "service", :name "test1", :new-field "some-value"}},
:random2
{:rng2
{:type "source",
:name "test2",
:some-other-field "some-other-value"}}}#2024-04-2515:58ambrosebs(m/validate [:multi {:dispatch D} [C1 S1] [C2 S2] [::m/default S3]] V)
=>
(({C1 (validator S1), C2 (validator S2)} (D V) (validator S3)) V)
so the keys of your clauses need to be = to the result of applying the dispatch function to the validated value.#2024-04-2516:02ambrosebsI think in your case you need to pull the :multi up to the top-level to wrap your :map if you want the dispatch to see those fields.#2024-04-2516:05ambrosebsActually maybe not, you just need to write a dispatch that returns either "service" or "source" depending on which branch you want.#2024-04-2516:06ambrosebsThat information seems to be in the dissoc'ed map.#2024-04-2606:31Tommi MartinThere was an issue in default usage of the schema (i think)
I've corrected it in the following:
(require '[malli.core :as m])
(defn test-dp
[m]
(clojure.pprint/pprint m)
#_"Should the dispatch return?"
"service")
#_"value of m in test-dp when both randomX fields are present:"
; {:random1
; {:rng {:type "service", :name "test1", :new-field "some-value"}},
; :random2
; {:rng2
; {:type "source",
; :name "test2",
; :some-other-field "some-other-value"}}}
(m/validate
[:map
[:name :string]
[:field1 :string]
[::m/default
[:multi {:dispatch test-dp}
["service" [:map-of
:keyword
[:map-of
:keyword
[:map
[:type :string]
[:name :string]
[:new-field :string]]]]]
["source" [:map-of
:keyword
[:map-of
:keyword
[:map
[:type :string]
[:name :string]
[:some-other-field :string]]]]]]]]
{:name "test"
:field1 "test-field"
:random1 {:rng {:type "service"
:name "test1"
:new-field "some-value"}}
#_:random2 #_{:rng2 {:type "source"
:name "test2"
:some-other-field "some-other-value"}}})
New addition in the schema was an addition map-of nesting at the top. If you remove the :random2 from the data to be validated and return a string service from the dispatch function, it works. But if both :random1 and :random2 are present it feels like the dispatch function breaks. as it gets a merged map in as a parameter. If the expected return is a string. The dispatch will not work with the schema described as it cannot return "random1 is a service" and "random2 is a source" at the same time.#2024-04-2606:56Tommi MartinIt looks like the solution I'm looking for here could be wrong.
Would anyone have examples of malli schemas that fulfil the following conditions:
1. The schema is for a map
2. The map has :keyword :string/double/int/boolean/etc. values
3. Map has atleast 2 keys that are change based on external factors and cannot be hardcoded into the map. The content of these keys is a map and it's structure does not change regardless or if the key changes?
4. The content of the submaps is validated#2024-04-2910:10Tommi MartinIt turns out i was using the multi and the default incorrectly. A working solution for this is as follows:
(require '[malli.core :as m])
(m/validate
[:map
[:name :string]
[:field1 :string]
[::m/default
[:map-of
:keyword
[:map-of :keyword
[:multi {:dispatch :type}
["service" [:map
[:type :string]
[:name :string]
[:new-field :string]]]
["source" [:map
[:type :string]
[:name :string]
[:some-other-field :string]]]]]]]]
{:name "test"
:field1 "test-field"
:random1 {:rng {:type "service"
:name "test1"
:new-field "some-value"}}
:random2 {:rng2 {:type "source"
:name "test2"
:some-other-field "some-other-value"}}})
Notice how the ::m/default syntax has changed. it no longer has multiple :map-of :keyword [:map-of :keyword [:map... values
I was way too tunnel visioned on keeping the declarations isolated. By having multiple map-of values in the the multi declaration and the dispatch. It caused malli to dump all of the remaining fields into the dispatch function. Placing the map-of values before the multi declaration changed the data sent to the dispatch function from what you see in previous messages to:
{:type "service",
:name "test1",
:new-field "some-value"}
And as expected the dispatch function is now called several times, once per each data object. With this, returning the string is possible.
Thank you @U055XFK8V for your help and sorry for taking your time with poor explanations.#2024-04-2910:24Tommi MartinSummarising the thread for my issue: it has been resolved. I now can return a string or a keyword from the dispatch function correctly. Problem wasn't the syntax of the dispatch function but the schema I had written. I used multi at the top of [::m/default [:multi ... which causes it to get all the remaining fields as a one big clump making correct return very difficult. In my case using [::m/default [:map-of :keyword [:map-of :keyword [:multi {:dispatch custom-db} ... syntax cleaned up the calls to dispatch from "one call that has all fields" to "multiple calls, once for each unresolved submap." and enabled the string/keyword return to work as intended.
I should've paid more attention to my schema over worrying about the nitpickings of how dispatch functions. Sorry for the bother.#2024-04-3001:15steveb8n@ikitommi I use Malli a lot in node.js but the error reporting is not great there. I guess this is because Metosin doesn’t use it in the node world much. Any chance we could see some improvements for node?#2024-04-3001:15steveb8nI hope that it’s not too much work since there’s already good tooling for cljs in browser (with devtools)#2024-04-3001:16steveb8nthe main problems are:#2024-04-3001:18steveb8n1. invalid schema doesn’t include location and error
2. invalid input/arity (fn schema) doesn’t include location and explain
3. stack traces from fn schema failures (via shadow-cljs) don’t refer back to source location#2024-04-3001:21steveb8nI’m using 0.16 and have enabled https://malli.cljs.dev/start! in my REPL but it doesn’t help like it does on jvm. jvm DX is super nice#2024-04-3001:35steveb8nI’d be happy to setup tests in the repo that reproduce the problems if you can confirm that someone will work on it?#2024-04-3007:15juhoteperiwith location, do you mean the source code file and line number?#2024-04-3007:17juhoteperiCljs can only access the source location in macros, so adding those to the error data could be really hard.
In JS side there could be some tricks, like throwing and catching an error and parsing the stacktrace...#2024-04-3007:18juhoteperiNot sure why the stacktrace wouldn't refer to the correct place. That could be feasible to fix.#2024-04-3008:02steveb8nYes. Source location is missing and would help a lot. Best would also include pretty error reports like jvm#2024-04-3008:05steveb8nI use try catch and other tricks to workaround atm#2024-04-3007:48ikitommi@ambrosebs thank you working hard on malli 🙇 My primary focus has been to get the reitit 0.7.0 out (releasing that now), so haven’t had time to go through all the draft PRs you are working on. After the releases, will go them through.#2024-04-3019:03ambrosebsThanks @ikitommi! none of them are fully baked, curious for your thoughts.#2024-04-3019:25ambrosebsThough I was pretty happy with :seqable and :-> , so maybe you can start there if you can resist the massive constraints PR :)#2024-04-3008:38ikitommi[metosin/malli "0.16.1"] pushed out. We rollbacked the dropping support of Java 1.8. Now tested with all LTS releases.#2024-04-3009:41ikitommi🥳#2024-04-3010:28vemvHow do I get the "primary" (simplest and most primitive) type of a Malli schema?
e.g.
• :string => :string
• [:and :string [:fn ,,,]] => :string#2024-04-3016:12geraldodevIt's probably expensive, but it's fun 🙂
(do (require '[malli.generator :as mg])
(->
; (mg/generate [:enum "s" "b" "d"])
; (mg/generate string?)
; (mg/generate :string)
; (mg/generate [:and boolean? [:fn identity]])
; (mg/generate [:tuple {:title "location"} :double :double])
(mg/generate [:map [:x int?] [:y int?]])
type))
It returns the class, need to transform that into the simplest and most primitive#2024-04-3016:19vemvI'll actually consider it :) thanks!
I wonder how the clj-kondo <-> malli integration does this (I reckon it has to do something similar to this)#2024-04-3016:21geraldodevI'm looking into generate, if you can tap into -generator maybe you skip the generate phase, and map the generator into to form. Just an idea#2024-04-3016:39geraldodevgenerators are implemented using multi methods, so that's not possible.#2024-04-3018:00geraldodevhttps://gist.github.com/geraldodev/6c69759db927c018e2d5f491f5f4be31#2024-05-0207:00ambrosebs@U45T93RA6 here's the clj-kondo magic. it just enumerates all the cases https://github.com/metosin/malli/blob/master/src/malli/clj_kondo.cljc#2024-05-1706:12opqdonutpotentially related: https://github.com/metosin/malli/pull/784#2024-05-0206:59ambrosebsSchemas for parametric polymorphism https://github.com/metosin/malli/pull/1053#2024-05-0521:38chromalchemyHow can I validate: map contains keys :a, :b but not :c, :d .
(can specify value schemas as needed)
is there a different key property than :optional?
(ps: claude ai recommended an :absent? property, but I see not docs for it and doesnt seem to work)
I think I do not want a closed map, because there are a bunch of optional keys that I want to allow (and would prefer not to necessarily specify them all) ?#2024-05-0521:49chromalchemyI was able to get it working like this
(m/schema
[:and
[:map
[:name :string]
[:type [:enum "physical" "digital"]]
[:weight positive-number]
[:price positive-number]]
[:fn {:error/message (str "product contains read-only keys")}
(fn [data]
(does-not-have-keys? data
[:id :date-created :date-modified :calculated-price :base-variant-id]))]]
)#2024-05-0521:50chromalchemyIs there a more built in schema logic syntax for that?#2024-05-0610:393starblaze[:and MAIN-SCHEMA [:map-of [:not [:enum :c :d]] :any] should do the trick and fail when :c and :d key is defined.#2024-05-0617:33chromalchemy Nice! Thank you#2024-05-0619:11ambrosebsthis PR adds first-class support for this (with robust humanizers and generators) https://github.com/metosin/malli/pull/1025
[:map
{:not [:or :id :date-created :date-modified :calculated-price :base-variant-id]}
[:name :string]
[:type [:enum "physical" "digital"]]
[:weight positive-number]
[:price positive-number]]#2024-05-0802:43lotucIf we got a never type or never property (which likes the optional), that would be convenient:
[:map [:a :never] [:b :some-type]]
;; or
[:map [:a {:never true} :some-type] [:b :some-type]]#2024-05-0803:43ambrosebs@U027DJ7CPPU there's one in malli.destructure/Never. To make it work with generators, use:
(defmethod mg/-schema-generator 'Never [_ options] (mg/-never-gen options))
#2024-05-0804:56lotucGreat! didn't notice that before.#2024-05-0714:54dvingoNot sure why I didn't think of this earlier since the browser UI is available in cljs apps! - I've been working on a DOM reporter for malli instrument with cljs:#2024-05-0714:57dvingothe code is here: https://github.com/dvingo/malli/blob/dv-mainline/src/malli/dev/dom_reporter.cljs
needs further work to support fully colorized output and all error types, but that's iterative work. The theme support is already in place.#2024-05-0804:54ikitommilooks great!#2024-05-1008:28arohnerIs there a malli<->spec compatibility layer anywhere? We have years of specs and I’d like to investigate migrating to malli without having to boil the ocean in a single PR#2024-05-1010:28valtteriAFAIK there’s not much in this field. Internally we’ve coined some ideas. One of them being generating example data from spec and using malli provider to infer schemas from the example data.
Another possible path could be some kind of a mechanical transformation. spec-tools may be helpful here.#2024-05-1010:28valtteriThird option is to use spec/malli side-by-side and gradually migrate over#2024-05-1010:29arohnerI’m mostly interested in the third. We have thousands of specs, so I think if malli implemented the spec protocols that would make a migration path#2024-05-1010:29arohner(s/spec (m/schema ...))#2024-05-1010:30arohner(s/gen (m/schema ..))#2024-05-1010:30valtteriInteresting, hadn’t thought about that#2024-05-1013:37Noah Bogartyou could use malli's validation functions in a spec:
(def IntSeq (m/validator [:sequential :int]))
(s/def ::temp IntSeq)
(s/valid? ::temp [1 2 3]) ; true
(s/valid? ::temp [:a 2 3]) ; false
#2024-05-1022:10ambrosebs@U050R7ECY I played around with your idea, it seems to work pretty well at least for validation. https://github.com/metosin/malli/pull/1057
;; example escaping back to spec from malli
(s/def :person/relatives (from/malli [:sequential
(from/spec :acct/person malli-options)]
malli-options))#2024-05-1010:03andrea.crottisuppose I have a config.edn file and I generate a malli schema for that, and a function that gets values from the config like (config/get [:some :path...})
I wonder if it would be possible to make the function aware of the schema and fail if you are trying to get something that should not be available#2024-05-1010:04andrea.crottiI guess if the config was valid according to the schema you should get a value back#2024-05-1010:05andrea.crottiwould be great if we could even check that at lint time with clj-kondo & similar#2024-05-1010:05andrea.crottibut maybe that's a bit too much to ask#2024-05-1214:06dominicmFun idea, I wonder if you could generate a schema from a config file. Assuming that your config file was written with something like Aero (so prod & dev live side by side).
Then you could check that your code follows the rules of reading from the config file in all conditions (e.g. dev returns a string, prod returns some kind of encrypted object)...#2024-05-1010:19arohnerSecond, is serializing malli schemas to a database supported / a good idea?#2024-05-1010:25valtteriIt’s a designed idea. 🙂 I mean, one of the goals of the design in malli is that schemas are data, so you can store them into a db, transfer over wire etc.#2024-05-1013:40Noah Bogartjust make sure that you write :fn schemas as data and either rely on SCI to evaluate them or eval them yourself (to be spicy)#2024-05-1318:31dvingoIn a clojure app using datomic I created the following malli schema:
(:import [datomic.db Db]))
(def DatomicDb (mc/-simple-schema {:type :datomic/db :pred (fn [v] (instance? Db v))}))
and that is working fine to validate a db instance.
I have a problem when using this type in a function schema - when there is an instrumentation failure the entire database is being serialized/printed in the instrument error. Is there a way to override how the pretty printer prints a certain type?#2024-05-1318:35dvingooh, was pretty easy:
(:require [fipp.ednize :as ednize])
(extend-type datomic.db.Db
ednize/IOverride
ednize/IEdn
(-edn [x] "
#2024-05-1319:40Ben SlessYou can also provide an error key to the schema#2024-05-1412:54dvingocan you provide an example, adding the :error key to -simple-schema has no effect.
The code path for the printer is (virhe/-visit) invoked here https://github.com/metosin/malli/blob/9f5b8095e1bc72b04ce8c006ae8951b73a0afcf9/src/malli/dev/pretty.cljc#L54
which calls fipp.visit/visit
https://github.com/metosin/malli/blob/9f5b8095e1bc72b04ce8c006ae8951b73a0afcf9/src/malli/dev/virhe.cljc#L131
and I'm guessing it's hitting the unknown branch here:
https://github.com/metosin/malli/blob/9f5b8095e1bc72b04ce8c006ae8951b73a0afcf9/src/malli/dev/virhe.cljc#L44
which calls the protocol fn fipp.ednize/-edn#2024-05-1406:57steveb8nQ: how experimental is malli.experimental.time.transform & malli.experimental.time? we’re tempted to use it in latest release#2024-05-1407:32juhoteperiThe API of the current namespace isn't likely to change. It is mostly experimental due to the JS solution which is tied to js-joda library.
If we were to change it considerably, I think we'd introduce a new time namespace.#2024-05-1409:03steveb8nCool. We're using cljc-java-time so we'll watch for issues when using it on node. Thanks#2024-05-1422:18steveb8nObservation: we love Malli and use it extensively. The feature that trips us up regularly is encoders/decoders. Any ideas or references for ways to navigate this feature?#2024-05-1422:20steveb8nwe have them working but they seem more likely to break than anything else. e.g. they work differently with :and vs :merge schemas but these seem like unrelated features#2024-05-1422:21steveb8nnot really complaining. just providing feedback and hoping for guidance. maybe this is an area that would be good for more docs? or maybe I’ll ask the fancy new gpt4o to explain it to me like I’m a 5 year old 😉#2024-05-1422:29Noah Bogartexamples would be helpful to know where the issues are#2024-05-1422:30Noah Bogarti believe :merge is a pain point currently, so i'm unsurprised it's causing you trouble#2024-05-1422:41steveb8nfair point. happy to provide examples but I’m more trying to highlight the part of the lib which is most difficult to use/debug. we have been able to figure it out each time but it takes longer than any other feature. thought it might be useful feedback to highlight that#2024-05-1521:53ambrosebsadded some basic docs for the schemas you mentioned specifically https://github.com/metosin/malli/pull/1058#2024-05-1523:34steveb8nlegend! could these examples be simplified i.e. not use interceptors but simpler transformers? I suppose not because these docs are illustrating the order of transform#2024-05-1600:28steveb8nthis makes me realise/refine how we experience trouble with encode/decode. we frequently notice that values are not transformed and it’s not obvious why. we fix by reducing the size of the registry/schema/value till we find the missing piece. I wonder if some kind of logging or other tooling could make this easier i.e. show hit/miss when walking down through data and applying transformers?#2024-05-1602:02ambrosebscould coerce help here? it returns the explained output of the failure on the partially decoded value (assuming you have a schema for the decoded value).#2024-05-1602:03ambrosebsI've never used transformers, if you can think of more helpful or realistic examples lmk#2024-05-1604:36Karol WójcikQ: How can we create a schema for Tuples (see https://docs.datomic.com/schema/schema-reference.html#composite-tuples) taking into account :db/valueType set to :db.type/tuple should imply either Composite Tuple, Heterogeneous tuple or Homogeneus tuple. Is there a syntax in malli to support this kind of linkage?#2024-05-1604:47Karol WójcikHope the answer is not multi schema with hand written enumeration of every combination 😂 #2024-05-1604:48ambrosebs;; implications of keys is not very well supported. can hack it together with :and+:map+:map-of.
[:multi {:dispatch :db/valueType}
[:db.type/tuple [:map [:db/tupleType {:optional true} ...]
[:db/tupleAttrs {:optional true} ...]
[:db/tupleTypes {:optional true} ...]]]]
This is directly supported in https://github.com/metosin/malli/pull/1025:
[:multi {:dispatch :db/valueType}
[:db.type/tuple [:map {:xor [:db/tupleType :db/tupleAttrs :db/tupleTypes]}
[:db/tupleType {:optional true} ...]
[:db/tupleAttrs {:optional true} ...]
[:db/tupleTypes {:optional true} ...]]]]
#2024-05-1604:54ambrosebs;; this is kind of hack I'm alluding to
[:multi {:dispatch :db/valueType}
[:db.type/tuple [:and
[:map [:db/tupleType {:optional true} ...]
[:db/tupleAttrs {:optional true} ...]
[:db/tupleTypes {:optional true} ...]]
[:fn #(= 1 (count (filter #{:db/tupleType :db/tupleAttrs :db/tupleTypes} (keys %))))]]
#2024-05-1605:05ambrosebsI wonder if I can directly add :dispatch as a keyset constraint like:
[:map {:dispatch [:db/valueType
[:db.type/tuple [:xor :db/tupleType :db/tupleAttrs :db/tupleTypes]]]
[:db.type/string ...]
[:db.type/ref ...]}
[:db/tupleType {:optional true} ...]
[:db/tupleAttrs {:optional true} ...]
[:db/tupleTypes {:optional true} ...]]
#2024-05-1606:23ambrosebsactually that works really well https://github.com/metosin/malli/pull/1025/commits/4f56c61d23138f1512adc3f27c37b9d65d2bdf7d#2024-05-1608:00Karol WójcikHm, what about chaining many :fn in :and? I think it might work really well to add schema contraints? What is your opinion? #2024-05-1615:17ambrosebsIMO it only works well for validation. it might make generators fail, and you need to add custom error messages. but it's your only choice atm (e.g., the "hack" schema I posted 3 messages ago).#2024-05-1615:59Karol WójcikThank you @U055XFK8V! I will patiently wait till this use case is more widely supported upstream! #2024-05-1614:41Ben WadsworthIs there/I can't find a way to apply a custom generator to a whole schema... ie. I have property A which I want to generator with wither a 1 or a 2 and then have property B generate with an X or a Y based on that generated A property.
(defn custom-generator
[]
(gen/let [prop-a (gen/elements [1 2])]
{:A prop-a
:B (if (= 1 prop-a) "X" "Y")}))
(def my-schema
[:map
{:gen custom-generator}
[:A [:enum 1 2]]
[:B [:enum "X" "Y"]]])
(mg/generate my-schema)
This doesn't work, I will get something like {:A 1 :B "Y"}#2024-05-1615:18ambrosebs{:gen/gen (custom-generator)}#2024-05-1615:20ambrosebsunfortunately I don't think it can be a fn? atm https://github.com/metosin/malli/blob/9f5b8095e1bc72b04ce8c006ae8951b73a0afcf9/src/malli/generator.cljc#L501-L505#2024-05-1615:29Ben Wadsworthfacepalm just had to use it as a function huh.. Thanks! I got what I needed#2024-05-1616:22ambrosebssome related discussion here https://github.com/metosin/malli/pull/1043#2024-05-1909:20Eugenhi, should this be an error?
(plantuml/transform [:schema {:registry r} "Network"])
; Execution error (IllegalArgumentException) at malli.core/-property-registry (core.cljc:258).
; Don't know how to create ISeq from: malli.registry$composite_registry$reify__10480
#2024-05-1915:57ambrosebsI think the rule of thumb is everything inside a schema needs to be serializable. so no IntoSchema definitions, composite schemas etc., (they should all go in options). related https://github.com/metosin/malli/issues/1048#2024-05-1909:20EugenI am trying to visualize a schema that also has :time/instant so I build a custom registry with defaults, time + my schema .
Also, I expected this to work but it does not
(plantuml/transform "Network" {:registry r})
; Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
; :malli.core/invalid-schema
while I can do
(mg/generate "Network" {:registry r})
#2024-05-1909:54Eugenseems like I can
(plantuml/transform :int {:registry r})
so I guess the issue is with my schema + plantuml#2024-05-1910:12EugenI made a reproducer for the issue I am getting with malli 0.16.1
(ns user
(:require [malli.experimental.time.generator]
[malli.experimental.time :as mt]
[malli.core :as m]
[malli.registry :as mr]
[malli.generator :as mg]
[malli.plantuml :as plantuml]
[malli.dot :as md]))
(def snm-schema {"Curve" [:enum :curve25519 :p256]})
(def r
(mr/composite-registry
m/default-registry
(mr/registry (mt/schemas))
(mr/registry snm-schema)))
(comment
(mg/generate "Curve" {:registry r})
;; => :p256
(plantuml/transform ["Curve"] {:registry r})
;; => Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
;; :malli.core/invalid-schema
(md/transform ["Curve"] {:registry r})
;; => Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:136).
;; :malli.core/invalid-schema
)#2024-05-1916:23ambrosebsApparently the registry needs to be embedded in the schema atm.
(println (plantuml/transform [:schema {:registry {"Curve" [:enum :curve25519 :p256]}} "Curve"]))
@startuml
entity Curve {
[:enum :curve25519 :p256]
}
@enduml
#2024-05-1916:30EugenI guess this is a bug ?#2024-05-1916:30Eugenshould I create an issue ?#2024-05-1916:30ambrosebsyeah probably#2024-05-1916:34ambrosebsmaybe ref handling needs an overhaul here. recursive schemas also don't work.#2024-05-1916:40Eugenmade an issue https://github.com/metosin/malli/issues/1060#2024-05-1916:50Eugen@ambrosebs: can you help me figure out how to build a schema from vars?
I have a bunch of and don't know how to build a schema from them .
It worked with a map {"Curve" [:enum ... }
(def Curve [:enum :curve25519 :p256])
#2024-05-1916:55ambrosebsLike this?
[:schema {:registry {"Curve" Curve}} "Curve"]#2024-05-1916:58Eugenwill this work if I have references?
(def KeyPair [:map {:title "Key pair"}
[:curve [:ref {:min 1 :max 1} #'Curve]]
and want to add KeyPair as well?#2024-05-1916:58Eugenit's a bit confusing honestly 🙂#2024-05-1917:00ambrosebsSo there's this separate feature for using vars as schemas, but I'm going to assume you want the simplest thing.
(def Curve [:enum :curve25519 :p256])
(def KeyPair [:map {:title "Key pair"}
[:curve [:ref {:min 1 :max 1} "Curve"]]])
[:schema {:registry {"Curve" Curve, "KeyPair" KeyPair}} ...]
#2024-05-1917:02Eugenyes, I would like to get this working - but was also curios to understand why / how it works and there are nota lot of docs#2024-05-1917:03ambrosebsHave you seen the docs on recursive schemas? https://github.com/metosin/malli?tab=readme-ov-file#recursive-schemas#2024-05-1917:04ambrosebsIt's a bit more advanced than you need, but the second example shows that whenever schema sees a registry reference, it eagerly replaces it with the schema it's pointing to.#2024-05-1917:09ambrosebsyou can see a bit of how it works by using m/deref, which replaces a ref with its mapped schema.
(take 3 (iterate m/deref (m/schema [:schema {:registry {"Curve" [:enum :curve25519 :p256]}} "Curve"])))
([:schema {:registry {"Curve" [:enum :curve25519 :p256]}} "Curve"]
"Curve"
[:enum :curve25519 :p256])
#2024-05-1917:12ambrosebsa dereferenced schema remembers the registry it was defined with
(-> [:schema {:registry {"Curve" [:enum :curve25519 :p256]}} "Curve"]
m/deref-all
m/options
:registry
mr/-schemas
(select-keys ["Curve"]))
{"Curve" [:enum :curve25519 :p256]}
#2024-05-1917:12ambrosebsthis is how you can have interdependent registries.#2024-05-1917:16ambrosebsregistries also use dynamic scope which is a bit tricky to think about but they work similar to dynamic vars in clojure. the most recently seen registry mapping wins when resolving a ref.#2024-05-1917:24Eugenthanks#2024-05-2310:34vemvAny hope with https://github.com/metosin/malli/issues/1051 ? We could really use this at work#2024-05-2921:44ambrosebsI think you can pass ::m/sci-options in your options map, which is passed to sci.core/init.
Perhaps (assoc options ::m/sci-options (assoc (m/-default-sci-options) :classes {:allow :all})).
Haven't tried it though.#2024-05-2921:05Joel[:or
[:map {:closed true} [:x :string]]
[:map {:closed true} [:a :string] [:b :string]]]
Hopefully, this was the way to represent what I want? Or, is there a better way? Right now the error message is funky, but I don’t mind customizing it. I just wanted to make sure this was the right way to go, I tried putting the :or inside one map decl, but it didn’t seem to like that.#2024-05-2921:35ambrosebsfor faster validation do a :multi dispatching on the presence of :x#2024-05-2921:37ambrosebs[:multi {:dispatch (comp string? :x)}
true [:map {:closed true} [:x :string]]
false [:map {:closed true} [:a :string] [:b :string]]]#2024-05-2921:38ambrosebsassuming there's more than 3 possible keys it usually helps.#2024-05-3003:18jasonjcknThere's a non-trivial performance cost to SCI as I understand, its an interpreter, and can't compile, are there any options to serialize specs that still gives high performance? basically I'll have to call eval, no... (?) anyone have clean solutions here.#2024-05-3004:02Ben SlessAs long as you don't use serialized fn specs there's no overhead, just make sure to emit the worker objects from your schema once and not on every call (e.g. call validator)#2024-05-3004:04jasonjcknright, but i am asking about the fn specs#2024-05-3004:29Ben SlessDid you check the performance overhead? What's your performance budget? SCI isn't slow, it's rather well optimized for an interpreter#2024-05-3004:30Ben SlessAlso, what's your performance budget?#2024-05-3004:30jasonjcknita gotta be like at least 20x slower? i can benchmark it#2024-05-3004:31Ben SlessMeasure first#2024-05-3006:42borkdudeIf you want to serialize to the front-end, cherry is the more performant option since it is compiled to JS#2024-06-0523:14steveb8nQ: is anyone else using Malli in node.js/aws-lambda? pretty sure this is less common but it would be great to share tips if anyone is willing#2024-06-1019:47Samuel LudwigIf I wanted to malli/assert the validity of a mildly large spec (a map of approx 30 keys) at an average rate of 1-20 a second), would I be better off making a custom malli/asserter function (to be used like one would a validator or coercer), or would the performance of a flat malli/assert be acceptable?#2024-06-1019:48Noah Bogartis the current impl slow for you? or are you just worried up front?#2024-06-1019:48Samuel Ludwigjust upfront :^)#2024-06-1019:48Noah Bogartthe team has done a lot of work to make it pretty fast already, so maybe do some performance testing first to see if it's worthwhile#2024-06-1019:51Samuel LudwigVery very fair:malli:, i will give it a shot 🙂#2024-06-1117:13ambrosebsLooking at the impl, things to keep in mind are:
• optional keys are always looked up, so a spec of 30 optional keys will lookup an empty map 30 times
• closed maps iterate through both the spec keys and the entire map
• if you have an open map of required keys, it looks pretty quick#2024-06-1117:19Samuel Ludwigthankfully everything is open, but there are a good amount of optional keys; regardless, I should update and say that I just opted to create a (def assert (m/coercer <MySchema> nil nil)) in the my-ns.spec namespace#2024-06-1117:20Samuel Ludwig(my namespaces are structured in such a way (#C013B7MQHJQ) that this makes a degree of sense for me)#2024-06-1117:24ambrosebsoh I see, I think that's always a good idea#2024-06-1117:26ambrosebsI thought you were asking about the perf characteristics of validating maps.#2024-06-1117:26Samuel Ludwigoh that is definitely great information to have dont get me wrong, its appreciated ⭐#2024-06-1209:29lambdamHello everyone
I spontaneously tried to define local registries with namespaceless keywords, like so:
(ns my-ns
...)
(def local-registry
{:api-key :string
:values [:sequential :string]
:output [:fn other-ns/plop?]})
;; When using the local registry in a :map schema, I get the following ExceptionInfo when compiling the form:
;; :malli.core/register-function-schema
(m/=> foo
[:=> [:cat [:map {:registry local-registry}
:api-key :values]]
:any])
(defn foo [m] ...)
;; This compiles well with the return value of the :=> schema being defined only in local registry
(m/=> bar
[:=> {:registry local-registry}
[:cat [:map [:api-key string] [:values [:sequential :string]]]]
:output])
(defn bar [m] ...)
1. I saw in the documentation that local-registry keys should be namespaced keywords or strings. Why does this seem to work for the :=> and not for the :map schema?
2. Why not allowing namespaceless keys in local registries?
3. When exposing a local registry to a schema, does it scope this local registry to sub schemas? Example in [:=> {:registry local-registry} [:map ... does the :map schema "sees" the same schemas than the :=> schema, ie global registry + local registry? And thus would scopes of local registries be scoped like variables in Clojure in nested let expressions?
Thanks a lot#2024-06-1216:12ambrosebs1. your syntax seems off, please post the full error.
2. a local registry is scoped dynamically so it is effectively a global variable. namespace qualification mitigates some of the negatives of dynamic scope.
3. yes, it is seen by all subschemas. it's more like binding than let.#2024-06-1216:26ambrosebsI'm guessing local registries are special because they can be rebound dynamically by schemas themselves. Schema registry entries like :=> and :map are provided by the options map, and can't be shadowed (meaningfully) by the local registry, since local registries must be serializable and Schemas are not https://github.com/metosin/malli/pull/1026#2024-06-2015:37lambdamThanks @U055XFK8V
Sorry for late reply
1. This is the error that I get when providing a local registry to :map with non-namespaced keys :
1. Unhandled clojure.lang.ExceptionInfo
:malli.core/register-function-schema
{:type :malli.core/register-function-schema,
:message :malli.core/register-function-schema,
:data
{:ns habilitatem.server.handlers.student.orientation-wizard-v2,
:name student-orientation-wizard-get,
:schema
[:=>
[:cat [:map {:registry {:result :map}} :system/now :datomic/conn]]
:result],
:data nil,
:key :clj,
:exception #error {
:cause ":malli.core/invalid-schema"
:data {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :result, :form :result}}
:via
[{:type clojure.lang.ExceptionInfo
:message ":malli.core/invalid-schema"
:data {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :result, :form :result}}
:at [malli.core$_exception invokeStatic "core.cljc" 136]}]
:trace
...
But when I provide the local registry to the :=> schema I do not get this error, and only when using the => macro. When using metadatas and collect, I do get the error.
2 & 3 I think I don't get the subtleties of global variables and binding vs let
I was thinking more about the shadowing ability of let and the fact of exposing a schema in a scope like let exposes a variable in a scope.
2. "a local registry is scoped dynamically so it is effectively a global variable" : since a local registry can be defined inline, how is it comparable to a global variable?
2. again; "namespace qualification mitigates some of the negatives of dynamic scope": how bad would it be to redefine locally a schema?
Thanks a lot#2024-06-1716:14Samuel LudwigA continuation of the above, of sorts. I did end up trying to build a transformer to strip out map keys with nil values, and one that transforms empty strings to nil. I'm now running into some kind of ordering issue wrt transformer evaluation.
I'm trying to run the nilify-empty-string transformer before the strip-nils transformer, but regardless of the order that I have them in in the mt/transformer call, the strip-nils is run first. When I do two separate decodes, with the first as the nilify-string, and the second as the strip-nils, I do get the result I expect.
Do transformers for certain types i.e. maps get run before others i.e. strings?#2024-07-0814:08Samuel LudwigMade a gist of a transformer for stripping out optional keys who's value is nil, as I didn't see one anywhere else, this was formed largely by imitation and trial+error, so improvements/recommendations are welcome rich2
https://gist.github.com/samuelludwig/fc8241c2d8cdf39a1eef4912b0798e08#2024-07-0814:08Samuel LudwigMade a gist of a transformer for stripping out optional keys who's value is nil, as I didn't see one anywhere else, this was formed largely by imitation and trial+error, so improvements/recommendations are welcome rich2
https://gist.github.com/samuelludwig/fc8241c2d8cdf39a1eef4912b0798e08#2024-06-1808:38hanDerPederwhy isnt there a :number schema?#2024-06-1809:48steveb8nOne reason could be platform differences ie jvm vs JavaScript#2024-06-1809:49hanDerPederare there numbers where number? returns different results on jvm and js?#2024-06-1809:51steveb8nNot sure but we've noticed long values are handled differently#2024-06-1811:58valtteriThis works in many cases
(def number-schema [:or [:int] [:double]])
#2024-06-1812:00hanDerPederHi!
Yes, we're doing that. Was just curious 🙂
Seems that noone has stepped up to add this: https://clojurians.slack.com/archives/CLDK6MFMK/p1707901304113259?thread_ts=1707866937.701339&cid=CLDK6MFMK#2024-06-1812:01valtteriAh, that could be a good first PR for a new contributor 😉#2024-06-2022:20Martin MarianoIs there a convention to write malli schemas on pascal case? couldn't find anything on the web, but all examples on github are on that.#2024-06-2515:33dvingoIt's just a convention - I am guessing it was picked up from / influenced by
https://github.com/plumatic/schema
One of the benefits I've found from using pascal case is that I would otherwise have to append -schema to all my schemas to distinguish them from vars that contain data - as there would often be collisions. So the pascal case convention helps cut out that noise. And since pascal case isn't used in clojure aside from this and there are no static types using that naming convention helps with reading code#2024-06-2409:04Leo PoulsonHello everyone. Has there been any progress / known workarounds for instrumenting multimethods (`mx/defmethod` ) etc? I can see this has been discussed in the past#2024-06-2515:17vlad_pohHi I'm moving to malli coercion and run into an issue. See below
"errors": [
{
"path": [
0,
"created"
],
"in": [
0,
"created"
],
"schema": "string?",
"value": "2024-06-19T20:47:30",
"message": "should be a string"
}
],
and this is the relevant portion of ring router
:coercion (reitit.coercion.malli/create
{:error-keys #{:coercion :in :schema :value :errors :humanized}
:compile mu/closed-schema
:strip-extra-keys true
:default-values true
:options nil}
what am i doing wrong?#2024-06-2515:22Samuel LudwigI think we might need some more information, like, what data you're hitting your endpoint with, what the spec is for the endpoint, when exactly this error happens, etc.#2024-06-2519:40millettjonIt looks like your schema might be expecting something like:
[{"created" "2024-06-19T20:47:30"}]
but got just the plain value "2024-06-19T20:47:30"??#2024-06-2520:02branch14What is the canonical way to "schema" a known string? Currently I use an :enum with only one option, but this feels hacky. I could use a regex, but this feels even more wrong. Example:
[:or
[:map
[:type [:enum "type-1"]]
[:type-1-prop :any]]
[:map
[:type [:enum "type-2"]]
[:type-2-prop :any]]]#2024-06-2521:27delaguardo[:= "type-1"]#2024-06-2604:54branch14Thx! Kinda obvious now.#2024-06-2520:48MiaHello Malli heroes! Cross posting from the reitit channel in case this one gets 👀 more often.#2024-06-2520:48MiaI'm having a problem with malli schema coercion. I want to use a schema from malli.experimental.time, but we don't have a malli schema registry in the project. We're using reitit with the default coercion setup from the docs: https://github.com/metosin/reitit/blob/master/doc/coercion/malli_coercion.md#2024-06-2723:41millettjonDoes it work if you set the default registry for your project?
We have this:
;; Set the default registry for the whole application.
(m.r/set-default-registry!
(m.r/composite-registry
(m/default-schemas)
;; Allow using vars as schema references.
;; See:
(m.r/var-registry)
;; java.time schemas
;; See:
(m.time/schemas)
;; joda time schemas
joda/schemas))
#2024-06-2723:41millettjonThat seems to work with reitit coercion for json requests.#2024-06-2723:42millettjonWe have some legacy code that uses joda so added some custom schemas for that as well.#2024-06-2811:15ikitommireitit + malli needs a good template-project (with this is it). I could update that (+ Java21 with Virtual Threads)#2024-06-2520:49MiaI'm getting invalid schema errors because it is trying to find the schema and can't, since it isn't registered
Is there a place for me to add it to the schema registry in, say, the :options key? What would that look like?#2024-06-2520:50MiaThe other thing is, it's just checking to make sure something is a timezone. I made my own predicate for this. Would there be a way for me to register a custom schema somewhere in the project that uses this predicate? Feels like it's the same issue (can't locate the registry) though#2024-06-2520:53ambrosebsHello Mia! In vanilla malli you'd make options like {:registry (mr/composite-registry m/default-registry (malli.experimental.time/schemas))}. According to the reitit docs, you might want to try passing that as your :options map.#2024-06-2520:58ambrosebsfor registering predicate schemas, here's a good example. you then assoc that into your registry under the name you'd like.
https://github.com/metosin/malli/blob/666757de06a2abeee960ef894d05098585032313/src/malli/core.cljc#L705#2024-06-2520:58ambrosebsfor example here's how the :any schema is added to the registry https://github.com/metosin/malli/blob/666757de06a2abeee960ef894d05098585032313/src/malli/core.cljc#L2422C1-L2422C22#2024-06-2612:52MiaThank you! I think this is exactly what I'm looking for!#2024-06-2808:54mkvlrhttps://github.com/borkdude/speculative are specs for clojure.core, is anyone aware of a similar project for malli?#2024-06-2811:04mkvlrthis this expected (malli.provider/provide {:foo :bar}) ;; => [:vector :keyword]?#2024-06-2811:12Ben SlessThe argument is a seq of samples#2024-06-2811:13Ben SlessSo provide iterates over the entries#2024-06-2811:13Ben SlessPass [{,,,} ,,,]#2024-06-2811:37mkvlr@UK0810AQ2 thank you!#2024-06-2823:57Nikolas PafitisIs there a fn in malli for getting the underlying schema from a parametric schema? For example from [:maybe :int] get :int and [:vector :string] get :string etc.#2024-06-2903:10escherizeHmm, what should it do for [:vector [:maybe :int]]?#2024-06-2906:59ikitommiChildren is a vector and you want the first child, so:
(-> [:maybe :int]
(m/children)
(first))#2024-06-2911:43Nikolas Pafitis@U051GFP2V it should return :int#2024-06-2911:44Nikolas PafitisBasically to return the underlying schema, might be from a :< or any other arbitrarily nested 'parametric' (don't know if this is the correct lingo) schema.#2024-06-2913:04escherizeI'm trying to understand what you want, what should [:vector [:maybe [:or :string :int]]] return?#2024-06-2916:38Nikolas Pafitisi don't know to be honest. Maybe [:or :string :int] or [:string :int] since that'd be the underlying schema of the element, either string or int.#2024-07-0315:19Samuel LudwigIs there anything extra I need to do in order to invoke the new :-> function schema for a defn schema in 0.16.2?
a la
(m/=> double-me [:-> :int :int])
(defn double-me [x] (* x 2))
repl is unhappy so far#2024-07-0317:18ambrosebs:-> is not part of the default registry, I'm guessing you need to add it.#2024-07-0317:21Samuel Ludwigoh? interesting, the release notes mention :-> explicitly being available, but yea i see a commit in the mentioned issue removing it from the registry#2024-07-0705:05esp1I've been adding malli schemas to my functions with :malli/schema https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schema-metadata, but it looks like https://github.com/metosin/malli?tab=readme-ov-file#clj-kondo configs are not getting generated from those when I call (mc/linter-config) or (mc/emit!). Is there a way to get malli.clj-kondo to consider function schema metadata?#2024-07-0709:13ikitommiHi, there is no automatic namespace collection happening with clj-kondo ns, you need to call malli.instrument/collect! first - it walks all loaded namespace and pulls the function schemas into the global registry. Clj-kondo picks them up from there#2024-07-0709:13ikitommithere is malli.dev/start! which does this all - collect, Instrument and emit clj-kondo#2024-07-0709:14ikitommiif the docs are missing this, contributions most welcome!#2024-07-1008:43lyallis it a bad idea to try to do "check that no user with this email already exists" type of validation with malli? If not, I'd appreciate any pointers on how to pass my datasource to the validation fn. Seems like the validators take options but I haven't been able to figure out exactly what those options can be used for just yet#2024-07-1010:12Stig BrautasetSimilar questions have come up in this channel a few times. I think it's not a good idea.
I would recommend restricting Malli to verify that the data is of the right "shape", not that you haven't seen that email before. (Nor that it's valid, for that matter, since only really attempting to send a validation email can do that.)
That said, if you have an entity with multiple email addresses in it, you can certainly check that all those are distinct from eachother with Malli.#2024-07-1010:20lyallseems reasonable but what's your rationale behind thinking it's a bad idea?#2024-07-1010:24Stig BrautasetPerformance cost is one reason, and difficulty of testing. https://clojurians.slack.com/archives/CLDK6MFMK/p1706115115762729?thread_ts=1706111434.068599&cid=CLDK6MFMK
There's also another thread here that you might find interesting: https://clojurians.slack.com/archives/CLDK6MFMK/p1691330835332889#2024-07-1203:06didibusDoing side effects in the validation predicate sound like folly. I wouldn't do anything that needs that in it.#2024-07-1205:58lyallI wouldn't say that doing a read operation from the db counts as a side effect, but I agree that this type of validation may be best done outside of malli#2024-07-1612:49dominicmIt might be neat to opt into it based on context.#2024-07-1114:32JoelThe friendly error messages, include the Value, Errors, and Schema section. For my errors, it’s a quite large registry (Schema section) that it’s printing. Is it possible to trim this to just the portion of the registry that’s relevant to the error, or barring that, just not show the Schema somehow?#2024-07-1114:46ikitommiDo you have an example of the error?#2024-07-1114:50Joel-- Validation Error ----------------- validation.test:444 --
Value
{:bar "Neither should I",
:foo "I shouldn't be here",
:then {:context {"some" "thing"}}}
Errors
["invalid type"]
Schema
[:schema
{:registry { ; 60 lines or so of the registry...
In this case, since the error seems to be “top-level” I suppose it makes sense to dump the whole schema, but if it’s some inner map, it would be nice if it were pared down.#2024-07-1115:01JoelIt’s validating against ::specs
{::spec [:map ...]
::specs [:or [:sequential ::spec] ::spec]}
In this case, it would be nice if just that part of the registry was given. The [“invalid type”] here is pretty vague, so I suppose I should be adding custom messages to the registry for that part.#2024-07-1115:43JoelAt this point, I’m leaning toward removing the Schema section if reasonably possible.#2024-07-1115:44JoelBTW, I realized the error is much better moving the [:sequential ::spec] to the end of the :or statement#2024-07-1116:06JoelI also realized I can override the defmethod and get what I want
(defmethod v/-format ...
#2024-07-1116:14Joel@U055NJ5CC Where is the location of the error being printed?
- validation.test:448 --
Since I use macros the location isn’t accurate, I’m looking to see if I can make that better.#2024-07-1121:03JoelFound that spot too, just had to “override”
:throwing-fn-top-level-ns-names
#2024-07-1208:00ikitommiFor large Schemas, I think we should have:
1. option not to present the schemas, we have this?
2. option to minify schemas
a. option to omit either registry or all props
b. option to hide irrelevant part of the schemas, same as there is … for values#2024-07-1208:01ikitommiIdeas and drafts most welcome, but, as you noticed, it’s a multimethod, so easy to change behavior in user space#2024-07-1516:10JoelI think the value the Schema section brings is to be able to see, oh, x isn’t supposed to be an int it’s supposed to be a string, something the errors section doesn’t seem to indicate. Maybe addressing that forgoes needing Schema section. I do think if there is a Schema it should default to your 2b option.#2024-07-1619:27ambrosebsHi @ikitommi, I have a one line bugfix that I found in my various experimentations, could you take a look? https://github.com/metosin/malli/pull/1071
EDIT: hmm I think this also applies to [:enum nil nil], I will add that fix too.
EDIT2: fixed nil enums, added docs explaining the gotcha.#2024-07-1713:11Noah Bogartone line bugfix> +63, -6#2024-07-1713:11Noah Bogartlol it's a good fix, just funny how that happens#2024-07-1715:19ambrosebsJust a quick fifty-liner!#2024-07-1620:58fvidesHi. I'm using malli for the first time in a greenfield project, and I think I've found a bug. Let's consider this snippet:
(ns malli-test.schemas
(:require
[malli.core :as m]
[malli.registry :as mr]))
;;; A mutable registry, spec-like
(def registry* (atom (merge (m/default-schemas))))
(mr/set-default-registry!
(mr/mutable-registry registry*))
;;; A helper function for registering new schemas
(defn defschema
[type schema]
(swap! registry* assoc type (m/schema schema))
type)
;;; Two example schemas, work fine
(defschema ::int-pair [:tuple :int :int])
(defschema ::int-pair-alias-ok ::int-pair)
(comment
;;; After evaluating the buffer:
;; Both schemas are in the registry
(m/schema? (get @registry* ::int-pair))
(m/schema? (get @registry* ::int-pair-alias-ok))
;; Can use then to validate things
(m/validate ::int-pair [0 0])
(m/validate ::int-pair-alias-ok [0 0])
;; *but* if I try to define a new schema with atributes (for documentation purposes, for example)
;; this always hangs for ever
(m/schema [::int-pair {:doc "Something"}])
)
Am I doing something wrong?? Thanks in advance.#2024-07-1621:36ambrosebsPerhaps try wrapping in :ref like (m/schema [:ref {:doc "Something"} ::int-pair])#2024-07-1621:38ambrosebsI believe if you put a keyword as the first element of a vector, it's treated like a schema constructor, such as those returned by (m/default-schemas)#2024-07-1621:40ambrosebsI just happened to fix the infinite loop here, I'll add your test case https://github.com/metosin/malli/pull/1069#2024-07-1709:56fvidesMany thanks for your fast answer!
If I understand your fix correctly, it detects the infinite expansion and throws an error, so I guess the following line is not correct conceptually:
(m/schema [::int-pair {:doc "Something"}])
In the documentation, in https://github.com/fvides/malli?tab=readme-ov-file#vector-syntax , states that vector syntax can be used as:
type
[type & children]
[type properties & children]
So I assumed that my declaration would be valid on the third case, which seemed useful because I could define a base generic type (for instance a string with a fixed width) and define others aliasing this one, adding properties (for instance documentation). Am I wrong? If this is the case, is the use of :ref the right way? I hadn't considered the :ref because the documentation states that is useful for recursive schemas, which is not the case.#2024-07-1715:06ambrosebsThe syntax is subtle here. There are two kinds of "reference" schemas. One is the documented :ref which is for recursive schemas. The other is created when you provide a qualified keyword.
::int-schema expands internally to [::m/schema {:id ::int-schema} [:tuple :int :int]]. This is something you never have to care about (called a "pointer" in the implementation that just prints as ::int-schema), but it illustrates why [::int-schema {:doc ""}] doesn't work. You need an actual m/IntoSchema implementation as the first part of the vector#2024-07-1715:07ambrosebsyou could also try [:schema {:doc ""} ::int-schema]#2024-07-1715:11ambrosebssyntax like "Reference" , ::reference , and #'Reference are called references in the implementation and have similar rules (they are sugar for ::m/schema).#2024-07-1715:12ambrosebsI'll write this up in a tutorial, please ask any follow-ups.#2024-07-1716:35ambrosebsNow that I look into this, maybe your syntax can be supported with a little tweak.#2024-07-1717:55ambrosebsThis PR unifies the two syntaxes for references and schemas so that [::foo {..props..}] works when ::foo is a schema or a reference https://github.com/metosin/malli/pull/1072#2024-07-1718:05ambrosebsI abandoned my tutorial in preference to the fix. If you could try it out I'd be grateful.#2024-07-1718:55fvidesHello, I've tried your fix, but I'm afraid it still hangs when eval the last form in the previous example.
Just to be thorough, I've replaced the "oficial" malli dependence with your dependence like this:
,,,
:deps {org.clojure/clojure {:mvn/version "1.11.1"}
#_#_metosin/malli {:local/root "../malli.git"}
io.github.frenchy64/malli {:git/sha "88b41ff8f91e2898953b29e9b0813fab9cd172bb"}}
;;;
I've checked that it includes your changes, restarted the REPL and evaluated the ns. When I arrive to the last form, it hangs. I've debugged with CIDER, it gets stuck in the malli.core/schema method, after calling the function -lookup-into-schema at 2183, then enters again and is stuck forever at line 2172.
On the other side, [:schema {:doc ""} ::int-schema] works fine 🙂
Thanks a lot#2024-07-1719:00fvidesPlease let me know if I can do something else to help you debug this.#2024-07-1719:36ambrosebsPlease try https://github.com/metosin/malli/pull/1072/commits/07373ad5188baa411466d069e87f07673980138a#2024-07-1719:38fvidesNow it works!!!#2024-07-1719:40ambrosebsI need to fix the printing, should print as [::int-schema {:doc ""}] but prints as [:schema ...]#2024-07-1720:08ambrosebsPlease let me know if this prints the schemas as expected: https://github.com/metosin/malli/pull/1072/commits/8eadc266e29629e92a9f7c2f132b4c60387d14aa#2024-07-1807:46fvidesYes, it does.#2024-07-1717:55ambrosebsThis PR unifies the two syntaxes for references and schemas so that [::foo {..props..}] works when ::foo is a schema or a reference https://github.com/metosin/malli/pull/1072#2024-07-1818:43dominicmIs there a way to get malli's coercion to handle sequential query params? e.g. client=1 as well as client=1&client=2 for [:map [:client {:optional true} [:sequential [:int]]]]? So far it only works for the latter, and I can't figure out a placement of :decode/string that would let me do the coercion to a sequence if I needed to. (Maybe a reitit question?)#2024-07-1818:47dominicmI got it working using a custom transformer:
(malli.transform/transformer
{:name :->collection
:decoders {:sequential (fn [x] (prn ::x x) (if (sequential? x) x [x]))}
:encoders {}}
malli.transform/string-transformer)
Which is perhaps candidate for inclusion in reitit itself for query params?#2024-07-1818:50ambrosebsRelated https://github.com/metosin/reitit/issues/298#2024-07-1818:51ambrosebsThis comment makes a similar transformer https://github.com/metosin/reitit/issues/298#issuecomment-1161945435#2024-07-1818:52dominicmLooks like there's some need/desire to be able to prevent this from applying to forms/headers/paths/fragments#2024-07-1818:52dominicmSo I'll probably put in parameter coercion override in as well for :query#2024-07-1820:29DrLjótssonI found a way to this without having to mess with the transformers https://clojurians.slack.com/archives/C7YF1SBT3/p1699435964119069?thread_ts=1680893999.997339&cid=C7YF1SBT3#2024-07-1820:30dominicmOh neat, of course#2024-07-1820:30dominicmThat's great#2024-07-1902:58ambrosebsCould someone help me document this, I've seen this question several times. here's a start, is this it? https://github.com/metosin/reitit/pull/690/files
Probably should also go here https://cljdoc.org/d/metosin/reitit/0.7.1/doc/ring/pluggable-coercion#2024-07-2220:25ambrosebsHi, I'm live streaming a talk August 7th on Malli https://youtube.com/live/y30fUDvEcnc?feature=share#2024-07-2609:01Jonathan BennettI asked this on #C053AK3F9, before I realized there was a malli channel. I'm guessing I'll get better help for it here.#2024-07-2615:45escherizecan you show us what sort of input and output you're trying to achieve?#2024-07-2622:27Jonathan BennettYes, here's the header row and a valid input row
MUL ID Chassis Model Role Type Size Movement TMM Armor Structure Threshold S S* M M* L L* E E* Overheat Point Value Abilities Front Arc Left Arc Right Arc Rear Arc
3563 Wolfhound WLF-2 Striker BM 1 12\" 2 4 3 -1 3 FALSE 3 FALSE 1 FALSE 0 FALSE 0 28 ENE, REAR1/1/-
And here is its output:
{:e* false, :left-arc "", :e 0, :structure 3, :rear-arc "",
:size 1, :model "WLF-2", :overheat 0, :armor 4,
:chassis "Wolfhound", :role "Striker",
:full-name "Wolfhound WLF-2", :s* false, :tmm 2,
:point-value 28, :s 3, :mul-id 3563, :l 1,
:movement {:walk 6}, :l* false, :right-arc "",
:m* false, :threshold -1, :front-arc "", :m 3, :type "BM",
:abilities "ENE, REAR1/1/-"}
EDIT: condensed output data structure.#2024-07-2622:29Jonathan BennettI'd also love it if the keys were in the same order and, as I understand it, Malli might also be able to help with that?#2024-07-2620:01ambrosebsPoll: What do you think about malli's scoping rules here?
(validate
[:schema {:registry {::Of :int
::Tuple [:tuple ::Of]}}
[:schema {:registry {::Of :boolean}}
::Tuple]]
[1])
;=> false
> 👍 if you want this behavior
> 🤷 if you have no strong opinions or just want to vote
> 👎 if you don't want this behavior
> 😮 if this surprised you#2024-07-2620:03Noah BogartThe registry is merged and then references are resolved?#2024-07-2620:05Noah BogartI would expect the registry to create the schema immediately, so creating a new version of ::Of would change nothing about ::Tuple, which was already created with the existing ::Of schema#2024-07-2620:05ambrosebsLocal registries use dynamic scope.#2024-07-2620:06Noah Bogartbrutal lol#2024-07-2620:06ambrosebsMy hypothesis is that almost no-one wants this or even knows about this.#2024-07-2620:08Noah BogartYeah, to me they're let-bindings. For example, consider this pseudo-code
(let [registry {::Of :int
::Tuple [:tuple ::Of]}
(let [registry {::Of :boolean}]
::Tuple))
would return [:tuple :int]#2024-07-2620:08Noah Bogart(obviously not real clojure code, many things wrong there lol)#2024-07-2620:09ambrosebsright but you're expecting registries to work like let. that's the intuition I'm expecting, and it's wrong.#2024-07-2620:09Noah Bogartyeah, I see that now.#2024-07-2715:05Vincent CantinIf it was local registries with lexical scope, then I would want this behavior.
In my experimental Malli-like lib, https://github.com/green-coder/minimallist/blob/all-work-and-no-play/test/minimallist/core_test.cljc#L284-L290`let` so that its semantic was obvious to all Clojurians.#2024-07-2715:12Vincent CantinThe reason I would want local registries with lexical scopes would be for cases where we want to compose small models into bigger ones.#2024-07-2819:35delaguardoI used that behavior once for generating malli schemas from typescript-like specification of elastic search API. however, this was not very human readable.#2024-07-2901:36ambrosebs@U04V4KLKC did the specification material use dynamic scope too?#2024-07-2909:17delaguardo@U055XFK8V no, it was static json spec with custom format. But it was easier to use dynamic scope in generated schemas.#2024-07-2916:21ambrosebs@U04V4KLKC was there any (mutual) recursion in the generated schemas? i.e., was :ref used?#2024-07-2916:24delaguardoyes, a lot :) I can share this experiment later today if you are interested.#2024-07-2919:42ambrosebsit would be useful for backwards compatibility purposes. I have some ideas on how to compile recursive validators more efficiently but dynamic scope makes it subtle.#2024-08-0616:18ambrosebsI talked to Kirill and he used dynamic scope to simulate generic interfaces translated from Typescript. If malli supported type functions, you'd write:
[:schema {:registry {::TupleOf [:tfn [:X] [:tuple :X]]}
[::TupleOf :int]]
But you can kind of simulate this by using dynamic scope:
[:schema {:registry {::Of :never
::Tuple [:tuple ::Of]}}
[:schema {:registry {::Of :int}}
::Tuple]]
The alternative would be to make your own IntoSchema.#2024-08-0617:05Noah Bogartthat's a clever workaround#2024-08-0913:06ikitommiMissed this thread. To be honest, I don’t recall what I was thinking here. Was thinking about (typed) holes, but looking at the current status, surprised how it works now 🙂 @U055XFK8V as you are working with both polymorphic schemas and with recursion, what is your current recommendation? As a consideration, both maps and multi support optional lazy refs, so would work weird in those cases anyway?#2024-07-3122:27SamI don't understand the difference between these two scenarios, when evaluating function schemas.
Starting with this code:
(ns myusername.mynewapp
(:require [malli.core :refer [=>]]
[malli.dev :as dev]
[malli.dev.pretty :as pretty])
(:gen-class))
(dev/start! {:report (pretty/thrower)})
(=> foo [:=> :cat :int])
(defn foo [] 3)
(foo) ;;works
Now, change the function spec and evaluate just the spec.
(=> foo [:=> :cat :string])
In this scenario, evaluating the function spec updates everything as expected.
However, if I change it back to the original and evaluate the original spec again, it crashes even if the function spec is correct. In this case, the registry was not updated?
{:tag :code, :attrs nil, :content ("(=> foo [:=> :cat :int]) " {:tag :_-, :attrs {:just "just", :be "be", :defn "defn", :this "this", :_ "_", :should "should", :string "string", :_3 "_3", :with "with", :foo "foo", :code "code", :crashes "crashes", :still "still", :eval "eval"}, :content nil})}{:tag :code, :attrs nil, :content ("\nTo fix it, I have to eval the defn which makes everything work again.\n\nHow come this is happening?")}{:tag :code, :attrs nil, :content nil}#2024-08-0103:22ambrosebs#2024-08-0107:08Sam#2024-08-0119:20ambrosebs#2024-08-0119:24ambrosebs#2024-08-0119:29Sam#2024-08-0119:32ambrosebs#2024-08-0119:33Sam#2024-08-0119:34ambrosebs#2024-08-0119:34Sam#2024-08-0506:47ikitommi#2024-08-0315:58maxtI noticed something that might be a bug. When I do () it writes this file .clj-kondo/metosin/malli-types-clj/config.edn :
{:report #object[malli.dev.pretty$reporter$fn__7322 "0x4e2e7a7c" "malli.dev.pretty$reporter$fn__7322@4e2e7a7c"],
:linters {:unresolved-symbol {:exclude [(malli.core/=>)]},
:type-mismatch {:namespaces {user {foo {:arities {1 {:args [:int], :ret :int}}}}}}}}
The #object part makes clj-kondo unhappy:
% clj-kondo --lint src/user.clj
WARNING: error while reading /Users/max/code/clj-scratch/.clj-kondo/metosin/malli-types-clj/config.edn (No reader function for tag object)
#2024-08-0316:01maxtSeems to be a regression because 0.16.1 doesn’t write the :report key at all. The error was in 0.16.2#2024-08-0316:22maxtIt’s also present in the current main. I filed a bug https://github.com/metosin/malli/issues/1083#2024-08-0506:47ikitommifixed in master, thanks for reporting!#2024-08-0506:48maxtThank you!#2024-08-0316:34maxtI like the new :-> syntax. It got me wondering if it would be possible to infer it, e.g. if the metadata is on a function, assume the list is arguments followed by return value
(defn foo
{:malli/schema [:int :int :int]}
[a b]
(+ a b))
I can’t imagine a schema for a function ever beginning with anything but :-> (or :=>) but maybe I’m wrong?#2024-08-0401:23ambrosebsthere might be a fun future with polymorphic functions [:all [:x] [:-> :x :x]]. (I'm a fan of infix: [:int :int :-> :int] ). referring the function type from a ref might also be another thing conflicting with this idea.#2024-08-0506:50ikitommiinfer as in malli.destructure/infer?
(defn foo [a b]
(+ a b))
(malli.destructure/infer #'foo)
; => [:=> [:cat :any :any] :any]
#2024-08-0506:56ikitommiand this works already:
(defn foo
{:malli/schema [:-> :int :int :int]}
[a b]
(+ a b))#2024-08-0506:59maxtNo, not as in md/infer, I just meant to skip :-> entirely it’s in a function context.#2024-08-0506:59ikitommiinfix is great, but it would be more syntax for the same thing and would not a valid schema.
{:malli/schema [:-> :int :int :int]}
vs
{:malli/args [:int :int :-> :int]}#2024-08-0507:00ikitommiI hear you, but saving the :-> means third way to define schemas. having all three ways in a codebase doesn’t make it simpler?#2024-08-0507:00maxtNo I realize it’s kind of a weird exception#2024-08-0507:01ikitommialso, ambiguity:
[:=> [:cat :tuple :int] :int]
[:-> :tuple :int :int]
[:tuple :int :int]
#2024-08-0507:06maxtI like the attaching the schema via metadata because it’s still works without malli, in contrast to using a defn wrapper. But it’s still wish it was a little more succinct. That’s why I was happy about the introduction of :->#2024-08-0507:06maxtAnd the reason why I started to think of ways of making it even shorter#2024-08-0516:40didibusIt might be :malli/schema that's too long?
What if there was meta keyword :malli/->
If you alias to m you even get:
m/-> [:int :int :int]#2024-08-0507:16ikitommicarved a release of all the ready stuff 🥳#2024-08-0508:20maxtJust tested 0.16.3 with :-> and ! and it works like a charm!#2024-08-0515:21yannvahalewynHi. I have a map with an optional key. I want empty field / blank strings sent from the client to result in the removal of that key. I feel like this is a common occurence, but using Reitit and Malli it gets pretty complicated so I'm wondering if there's an easier way.
Here's what I have done so far:
Custom transformer
;; A Schema with string decoder to test for blank strings
(def schema
[:map
[:my-key {:optional true
:decode/string #(when-not (str/blank? %) %)}
:string]])
;; This nilifies the blank string correctly
(m/decode schema {:my-key ""} (mt/string-transformer)) ;; => {:my-key nil}
;; A custom transformer for stripping optional nil keys
(defn strip-optional-nil-keys-transformer
"A Malli transformer that removes optional keys that have `nil` values from
maps. This is useful to coerce structures prefering `:optional` keys over
`:maybe` values."
[]
(mt/transformer
{:name :strip-optional-nil-keys
:decoders
{:map
{:compile
(fn [schema _]
(when-let [optional-keys
(->> schema
(m/entries schema)
(filter (comp :optional m/properties second))
(map first)
(seq))]
{:leave ;; On leave so that things can decode to nils first
(fn [record]
(if-let [remove-keys
(seq (filter #(nil? (get record %)) optional-keys))]
(apply dissoc record remove-keys)
record))}))}}}))
(m/decode schema {:my-key ""} (mt/transformer
(strip-optional-nil-keys-transformer)
(mt/string-transformer)))
;; => {} Great, works!
I have solved it by having two transformers, one from blank string to nil, another to remove optional keys with nil.
How would you solve this?
Adding transformer to
I need to use these transformers using reitit.coercion.malli . I couldn't find any documentation to achieve this, and following the code was non trivial. Here's how I got it working:
;; A reitit handler map
{:coercion
(reitit.coercion.malli/create
(assoc-in reitit.coercion.malli/default-options [:transformers :string :default]
(mt/transformer
(mt/strip-extra-keys-transformer)
(mt/string-transformer)
(strip-optional-nil-keys-transformer) ;; My custom transformer
(mt/default-value-transformer))))}
I find it unweildy to have to assoc into the nested default-options structure and re-create the list of default transformers in order to add mine. I'm also unsure how the :string and :body keys in :transformers is used, by trial and error :string worked but ¯\(ツ)/¯. Can anyone provide me with some more info?#2024-08-0611:33SamMaybe malli select can help you? https://github.com/eval/malli-select#2024-08-0709:38yannvahalewynHow would malli-select help?#2024-08-0709:42SamHmm, I guess I thought malli-select would be useful for generating the schemas, but I realize now that your issue is adding it to coercion.malli where it might not be as useful.#2024-08-0710:03yannvahalewynIndeed there's no issue generating the schemas. But thank you for the suggestion 🙂#2024-08-0518:49Serge LernerHi, is there a tool to generate malli schema from existing openapi spec?#2024-08-0608:22Serge Lerneri tried openapi-generator which support clojure but works with clojure.spec#2024-08-0619:00fuadIs there a recommended way to generate clj-kondo type specs from malli instrumentation during a CI run?#2024-08-0912:53ikitommiI don’t think there is. What would you do with the generated specs in or after the CI run?#2024-08-0912:57fuadI'm still experimenting with the instrumentation feature, so I might have some misconceptions, but as I understand, for clj-kondo to properly lint an ns containing functions defined via mx/defn, it needs the file that lives at .clj-kondo/metosin/malli-types/config.edn. I currently have the entire .clj-kondo/ dir, with the exception of the kondo config itself, ignored in git. So my thinking is I need to generate those files in CI in order for clj-kondo to run.#2024-08-0912:58fuadOn CI, kondo runs with --copy-configs, so that I wouldn't have to check in the configs into the repo.#2024-08-0717:24ambrosebs#2024-08-0809:30ikitommired onions, will watch in full later today 🙂#2024-08-0721:11RyanIs there a good pattern for making this schema less verbose? specifically relating to the shared elements of the multi-schema: (def test-schema [:multi {:dispatch :type}
[:ns/type-a [:map [:id 'string?] [:type ['qualified-keyword? {:namespace :ns}]] [:details 'nil?]]]
[:ns/type-b [:map [:id 'string?] [:type ['qualified-keyword? {:namespace :ns}]] [:details 'int?]]]
[:ns/type-c [:map [:id 'string?] [:type ['qualified-keyword? {:namespace :ns}]] [:details 'string?]]]
[:ns/type-d [:map [:id 'string?] [:type ['qualified-keyword? {:namespace :ns}]] [:details 'any?]]]
])```
#2024-08-0721:11Ryanthe only part that differs is the schema of the :details key, the rest is identical for every type#2024-08-0803:11exitsandmanI don't use Malli, but that schema looks like Just Data™ to me. You can probably use a for with a case for the different bit - if you really want to avoid ~15 lines of repeated code, that is#2024-08-0811:02ikitommiif you have ideas on how to make a good data syntax to simplify this, I’m all ears. Have a old draft of optionally supporting :extends property for maps. Not sure if that is a good idea…#2024-08-0814:13RyanI’m not sure I have any concept of how feasible/reasonable something like this would be, but it would be great if I could just use the multi-schema dispatch on the [:details…] .. there’s probably complexity in having the dispatch value be a sibling property and not a child of the [:details…], but the pattern shows up often enough in the work I do that i’d love a cleaner way to achieve it.#2024-08-0816:36ambrosebsa direction to look into might be to enhance :merge with the ability to handle :multi . Maybe when you compile a validator for such a schema, the dispatch/s of the child multi's are pushed up to the top level.#2024-08-0816:38ambrosebs[:merge
[:map [:a :int]]
[:multi {:dispatch :bar}
[:baz [:map [:bar :baz]]
[:flub [:map [:bar :flub]]
this would compile a validator similar to
[:multi {:dispatch :bar}
[:baz [:map [:a :int] [:bar :baz]]
[:flub [:map [:a :int] [:bar :flub]]
#2024-08-0816:39ambrosebsyou'd push the merge into the multi as the intermediate step:
[:multi {:dispatch :bar}
[:baz [:merge [:map [:a :int]] [:map [:bar :baz]]]
[:flub [:merge [:map [:a :int]] [:map [:bar :flub]]]
#2024-08-0816:41ambrosebs~this might event work for parsing, I don't think ~
:multi clauses are surfaced in the result of m/parse so we'll need to figure out a stable way to do that. e.g., :multi 's are nested left-to-right, [:merge :dispatch1 :dispatch2 :dispatch3] => [:dispatch1 [:dispatch2 [:dispatch3 :merge]]]
I'm guessing we'd build a protocol for a "mergable" type that multi and others would extend (maybe :or?).#2024-08-0817:41ambrosebsfeels a bit more general than mergable, maybe distributive.#2024-08-0818:30ambrosebsprototype https://github.com/metosin/malli/pull/1086#2024-08-0821:08ambrosebs@U020G0VEL75 could you try my branch and tell me if it's what you're after?#2024-08-0912:50ikitommiLove the approach, commented the issue.#2024-08-0913:12Ryan@U055XFK8V I will try that now, might take me a minute to figure out how to build it 🙂 relatively new to things off the beaten path with dependencies!#2024-08-0810:48ikitommimy quick poke on malli + OpenAI structured outputs. Simple and beautiful - https://x.com/ikitommi/status/1821495690314326076#2024-08-0810:55ikitommithought of adding malli.experimental.openai, but the needed glue code is just few lines of code, so just a gist: https://gist.github.com/ikitommi/e643713719c3620f943ef34086451c69#file-openai-clj#2024-08-0811:00ikitommiwrote https://github.com/wkok/openai-clojure/issues/66 to openai-clojure to add support for this.#2024-08-0901:45Nim SadehWhat's summer.openai? I currently use wkok's openai library#2024-08-0906:55ikitommiThe full code is in the gist linked above#2024-08-0906:56ikitommiwkoks library sadly didn't work here, tied to old openapi definition, thus my issue to update it#2024-08-0910:09simonkatzI have a spec something like this, specifying constraints on individual map entries and also on the relationships between different values in the map:
(def RelationshipBetweenXAndY
[:fn
{:error/message "bad relationship"}
(fn [{:keys [x y]}]
(= x y))])
(def MySpec
[:and
[:map
[:x :int]
[:y :int]]
RelationshipBetweenXAndY])
Is there a way to modify this so that the RelationshipBetweenXAndY part is only checked if the first part of the :and gives no errors?#2024-08-0912:51ikitommivalidation short-circuits already on first failing, right? This has been talked earlier, there might be issue about taking just first error from :and.#2024-08-1020:49VickySurajI am using malli.core/=> for function schema validation, I am able to handle standard fn-arguments with their respective validator-fn but how can I handle variadic arguments. A sample code
(malli.core/=> fn-with-variadic-args
[:=> [:cat {:malli/fn-name with-var-args} int? string? int?] any?])
(defn fn-with-variadic-args
[int-val string-val int-val' & [map-val]]
(prinlnt (format "%s:%s => {%s}" string-val (+ int-val int-val') (:string-value map-val))))
I am getting `Invalid function arity` exception staging expected 3 got 4#2024-08-1102:24ambrosebstry something like this https://github.com/metosin/malli/blob/39244905665541c8d895c8b121b6ecf508fc2b18/test/malli/experimental_test.clj#L103-L106